This feed does not validate.
... os.org/</link><language>en-US</language><author>Haiku Inc.</author><righ ...
^
... US</language><author>Haiku Inc.</author><rights>(C) 2001-2025 Haiku, Inc ...
^
... aiku, Inc. All rights reserved.</rights><updated>Thu, 10 Apr 2025 23:20: ...
^
line 1, column 500: (14 occurrences) [help]
... t_march_2025/</link><author>waddlesplash</author><pubDate>Thu, 10 Apr 20 ...
^
participation in this program.</p></description></item></channel></rss>
^
In addition, interoperability with the widest range of feed readers could be improved by implementing the following recommendations.
<rss version="2.0" xmlns:atom="https://www.w3.org/2005/Atom"><channel><title ...
line 553, column 993: (21 occurrences) [help]
... to it from an existing install!)</p></description></item><item><titl ...
^
<div class="highlight"><pre tabindex="0" style="color:#d0d0d0;backgrou ...
<div class="highlight"><pre tabindex="0" style="color:#d0d0d0;backgrou ...
<p><details><summary>What is a run loop?</summary>
participation in this program.</p></description></item></channel></rss>
^
<rss version="2.0" xmlns:atom="https://www.w3.org/2005/Atom"><channel><title>Haiku Project</title><link>https://www.haiku-os.org/</link><language>en-US</language><author>Haiku Inc.</author><rights>(C) 2001-2025 Haiku, Inc. All rights reserved.</rights><updated>Thu, 10 Apr 2025 23:20:00 -0400</updated><item><title>Haiku Activity & Contract Report, March 2025</title><link>https://www.haiku-os.org/blog/waddlesplash/2025-04-10-haiku_activity_contract_report_march_2025/</link><author>waddlesplash</author><pubDate>Thu, 10 Apr 2025 23:20:00 -0400</pubDate><guid>https://www.haiku-os.org/blog/waddlesplash/2025-04-10-haiku_activity_contract_report_march_2025/</guid><description><p>This report covers hrev58697 through hrev58787.</p>
<h3 id="applications">Applications</h3>
<p>jscipione did some minor code cleanups in Tracker.</p>
<p>humdinger added some more translator credits to AboutSystem.</p>
<p>OscarL adjusted ProcessController to display &ldquo;system resources&rdquo; before &ldquo;caches&rdquo; in the memory menu, making it much easier to understand what the numbers mean (and bringing it more in line with other items in the memory menu.) waddlesplash cleaned up the kernel memory accounting code in ProcessController to actually work properly, fixing some long-standing TODOs. He also did some other code cleanup around return types and string handling as well.</p>
<p>bitigchi made Terminal use the Unicode &ldquo;multiplication sign&rdquo; character instead of ASCII &ldquo;x&rdquo; in some more places (in the GUI).</p>
<p>AkashKumar7902 contributed a change to make the time tooltip on the seek slider in MediaPlayer display on hover, not just on drag. waddlesplash then made some more improvements to the tooltip to make it update much more smoothly. X512 added a &ldquo;25% scale&rdquo; video playback menu option.</p>
<p>apl merged a very large refactor of HaikuDepot to improve the application&rsquo;s performance and responsiveness.</p>
<h3 id="command-line-tools">Command line tools</h3>
<p>OscarL tweaked the <code>keymap</code> tool to show help with <code>-h</code>, and use <code>-H</code> for the &ldquo;header&rdquo; option instead.</p>
<p>humdinger rewrote <code>waitfor</code> to use a <code>BApplication</code>, and added an option to wait for the network being active.</p>
<h3 id="kits">Kits</h3>
<p>apl added a way to add and remove items from <code>BColumnListView</code> in bulk.</p>
<p>waddlesplash made some fixes to the tooltip management code, making it much more possible for controls specify specific tooltip positions and have them be respected. (This made the MediaPlayer changes work much more nicely.)</p>
<p>waddlesplash fixed a use-after-free discovered by the guarded heap in <code>BSlider</code>.</p>
<p>PulkoMandy added some missing macros to <code>elf.h</code> needed by WebKit.</p>
<h3 id="servers">Servers</h3>
<p>madmax fixed the build of the <code>test_app_server</code> setup after recent refactors.</p>
<p>korli fixed handling of failed condition checks in <code>launch_daemon</code>, implemented the &ldquo;file created&rdquo; event, and added support for &ldquo;driver settings&rdquo; file format for conditions (fixing some old bugs in the driver settings code along the way). He also fixed stopping services, and added some missing documentation comments.</p>
<p>nipos fixed some code duplication in <code>app_server</code> font rendering code.</p>
<p>madmax add some tests for <code>app_server</code> text rendering, and made some fixes to check glyph bounding boxes, correctly draw text with a clipping path, and text decoration lines being drawn with transforms.</p>
<p>augiedoggie added a <code>syslog_max_history</code> option to the kernel settings file, to control how many old syslog files are kept: the default is 1, but that default can now be increased if you want to keep more old syslogs.</p>
<h3 id="drivers">Drivers</h3>
<p>waddlesplash merged changes to the <code>iaxwifi200</code> driver (<code>iwx</code> on the BSDs) from OpenBSD to add support for another device.</p>
<p>waddlesplash merged changes to the <code>atheros813x</code> ethernet driver from upstream FreeBSD to fix operation on some newer devices.</p>
<p>waddlesplash tweaked the SCSI bus and drivers to use <code>NotifyOne</code> not <code>NotifyAll</code> on its condition variables. (This doesn&rsquo;t change behavior, because there&rsquo;s only ever one waiter on these variables, but it&rsquo;s important to be clear here because some other threads may destroy the variable immediately after being woken up.)</p>
<p>waddlesplash cleaned up the interrupts glue code in <code>ralinkwifi</code>, potentially fixing some bugs. He then imported a driver for <code>MT7601U</code> devices from FreeBSD, however it isn&rsquo;t clear if this driver actually works properly on Haiku yet.</p>
<p>waddlesplash added some logic to <code>ram_disk</code> missed in recent refactors, and fixed a hang in its &ldquo;TRIM&rdquo; routine.</p>
<p>korli implemented &ldquo;autonomous power save&rdquo; in <code>nvme_disk</code>, reducing power usage by ~1W on one of his systems.</p>
<p>X512 made some changes to input drivers to allow <code>input_server</code> to actually be interrupted/restarted.</p>
<p>X512 made the <code>framebuffer</code> driver not attempt to map the whole PCI BAR, but only just enough for the framebuffer itself. (This was a holdover from the VESA driver, which does want to do that for various reasons, but the dumb framebuffer driver should not.)</p>
<h3 id="file-systems">File systems</h3>
<p>waddlesplash added checks, first to <code>devfs</code> and then to many other filesystems, for the <code>O_DIRECTORY</code> flag on opening files, fixing <code>cp file /dev/some/block/device</code> among other things.</p>
<p>Jim906 contributed a patch to handle stale inodes much better in the NFS4 (client) driver, fixing a number of common kernel panics it caused, and also made inode fetching allowed to fail for removed nodes and fixed a locking problem in the read routine, fixing more kernel panics. He also wrote a number of debug output routines to help with diagnosing problems in the driver.</p>
<h3 id="libroot--kernel">libroot &amp; kernel</h3>
<p>waddlesplash refactored the kernel <code>Thread</code> structure to use the standard linked-list template classes, instead of manually managed ones, and switched it to use doubly-linked lists instead of singly-linked ones, making removals of thread structures O(1) instead of O(N). He also fixed a bug that was preventing correct <code>time</code> accounting for living child teams, a small memory leak in a corner case in thread destruction, and a TODO about when IDs wrap in <code>get_next_thread_info</code>.</p>
<p>waddlesplash made the <code>mutex_destroy</code> &ldquo;not owned&rdquo; assertion print a backtrace of the actually owning thread automatically.</p>
<p>waddlesplash added another invocation of <code>cpu_pause</code> in a spin loop in the condition variables code, to avoid burning CPU cycles unnecessarily.</p>
<p>waddlesplash adjusted the generic <code>memcmp</code> implementation to compare <code>size_t</code>s (which are 64-bit on 64-bit architectures) instead of <code>uint32_t</code>s, added the same multiple-bytes optimization to <code>strncmp</code> as <code>strcmp</code> already had, and avoided unnecessary subtractions in <code>memcmp</code>, <code>strcmp</code>, and <code>strncmp</code>.</p>
<p>waddlesplash cleaned up a lot of file organization and build rules for the &ldquo;strings&rdquo; code (<code>memcmp</code>, <code>strcmp</code>, etc.), especially the architecture-specific files. He then rewrote the generic <code>memcpy</code> and <code>memset</code> implementations (applying more optimizations to <code>memset</code> in particular), made the generic versions always used in the bootloader, and replaced the homegrown x86 <code>memcpy</code> and <code>memset</code> assembly implementations with NetBSD&rsquo;s (which seems to have been the upstream for FreeBSD and others.) (The homegrown x86_64 <code>memcpy</code> and <code>memset</code> appear to be better optimized than any of the BSD&rsquo;s implementations, so they remain as-is.)</p>
<p>waddlesplash fixed a race condition in the new FIFO implementation, fixing some lock-related KDLs. He also added an assertion to the common locking code that would&rsquo;ve made the problem easier to find.</p>
<p>waddlesplash fixed <code>write()</code> in libroot not handling values outside the 32-bit range properly on 64-bit architectures, and also fixed displaying such large values in <code>strace</code> properly (for all syscalls that return <code>ssize_t</code>, not just <code>read</code> and <code>write</code>, too.)</p>
<p>waddlesplash made a number of fixes to FPU context state handling in the kernel on x86_64, including resetting the state properly on all thread entries, storing the userspace FPU state in the kernel thread structure instead of the kernel thread stack (which is also a small performance optimization), sending the proper <code>FPE_*</code> code corresponding to faults with <code>SIGFPE</code>, properly handling control words on context switch, and more. This fixes a number of corner-case KDLs and application crashes.</p>
<p>waddlesplash adjusted the kernel to not report protection flags for kernel areas to any user except root, so that ProcessController can be sure it can make use of them. He also fixed the slab memory manager (which backs the kernel <code>malloc</code> and <code>object_cache</code>s) to report its areas as writable (which they always are), which fixes the memory counts for the kernel in ProcessController.</p>
<p>waddlesplash renamed many of the basic interrupt functions in the kernel to use &ldquo;intr&rdquo; (if not &ldquo;interrupt&rdquo;) rather than the abbreviation &ldquo;int&rdquo;, which could be confused with &ldquo;integer&rdquo;. (BeOS used &ldquo;interrupt&rdquo;, the abbreviation &ldquo;int&rdquo; was a holdover from NewOS.)</p>
<p>waddlesplash fixed &ldquo;LA57&rdquo; (5-level paging) mode support in the kernel to properly free its page tables, which were previously getting leaked. (There are some users running Haiku in KVM on systems with LA57 support now, so it&rsquo;s more important that this works than it was a few years ago.)</p>
<p>waddlesplash fixed a deadlock reported by korli in the filesystem caching code related to page wait behavior (found by running the <code>gVisor</code> tests on Haiku.)</p>
<p>waddlesplash improved handling of invalid pointers in the guarded heap, printing more verbose messages for them instead of just &ldquo;generic segfault&rdquo;.</p>
<p>waddlesplash made <code>dup3</code> return <code>EINVAL</code> if the old and new file descriptors are the same, bringing it in line with POSIX.</p>
<p>waddlesplash added <code>qsort_r</code> to libroot (it&rsquo;s in the newest version of POSIX.)</p>
<p>waddlesplash did a large number of cleanups to the remaining <code>glibc</code>-derived code in libroot (and also the legacy <code>libstdc++</code>): dropping private definitions from public headers, removing functions already defined elsewhere, deleting unneeded files, replacing more files with musl or BSD equivalents, adding the <code>stdio_ext</code> routines as found in musl and elsewhere (and needed by Gnulib after the removal of the private definitions), dropped many unused math files, dropped glibc locale structures and made it interface with Haiku locale structures directly, and much more.</p>
<p>waddlesplash upgraded <code>getopt</code> to the version found in glibc 2.41.</p>
<p>korli fixed the kernel <code>vsnprintf</code> to correctly print fractions smaller than <code>0.1</code>.</p>
<p>korli fixed the <code>create_dir</code> syscall to return <code>EEXIST</code> if a node already exists, before <code>EROFS</code>.</p>
<p>Anarchos made some improvements to the bootloader menu for PXE boot, displaying &ldquo;Network&rdquo; as the boot method, displaying the IP address of the boot volume, etc.</p>
<h3 id="build-system">Build system</h3>
<p>waddlesplash fixed a build regression on macOS by disabling some unneeded code on all platforms but Haiku in the Support Kit.</p>
<p>waddlesplash fixed the libroot stubs generator (a rarely-used tool that regenerates some files needed for bootstrapping) to work on Python 3.</p>
<p>korli made some minor changes to fix warnings from AddressSanitizer.</p>
<h3 id="thats-all-folks">That&rsquo;s all, folks!</h3>
<p>Thanks again to all who contribute to Haiku, and especially those donors who make my contract possible!</p></description></item><item><title>Haiku Activity & Contract Report, February 2025</title><link>https://www.haiku-os.org/blog/waddlesplash/2025-03-11-haiku_activity_contract_report_february_2025/</link><author>waddlesplash</author><pubDate>Tue, 11 Mar 2025 14:20:00 -0400</pubDate><guid>https://www.haiku-os.org/blog/waddlesplash/2025-03-11-haiku_activity_contract_report_february_2025/</guid><description><p>This report covers hrev58584 through hrev58696.</p>
<h3 id="applications">Applications</h3>
<p>jscipione fixed opening folders by double-clicking the &ldquo;Location&rdquo; column in query results in Tracker. He also fixed hard-coded values for certain menu item metrics, bringing them in line with standard menu metrics, cleaned up some parts of drag &amp; drop support, refactored some of the drawing logic, made some fixes to add-on loading, and more.</p>
<p>nephele removed Gopher from the declared list of supported protocols in WebPositive. (It was supported when we used the Haiku-native network backend, but now that WebPositive uses CURL again, it isn&rsquo;t supported.)</p>
<p>waddlesplash made some changes to HaikuDepot (inspired by patches from oco) to improve performance somewhat when searching for items.</p>
<p>captain0xff (a new contributor!) removed some duplicate code from Screen preferences.</p>
<p>waddlesplash fixed some memory-management problems and race conditions in Tracker, fixing at least one crash.</p>
<h3 id="command-line-tools">Command line tools</h3>
<p>korli added support to <code>strace</code> to dump the arguments of <code>rlimit</code> syscalls, and the &ldquo;type&rdquo; argument of <code>mmap</code> syscalls.</p>
<p>OscarL merged the &ldquo;filteredquery&rdquo; command-line tool into the &ldquo;query&rdquo; tool, meaning that &ldquo;query&rdquo; can now filter results by directory.</p>
<h3 id="kits">Kits</h3>
<p>jscipione implemented support for keyboard shortcuts without the <code>Cmd</code> key in the Interface Kit. (Previously all menu shortcuts, no matter what other modifier keys they used, were also required to have <code>Cmd</code> as one of them.) He then modified some applications to make use of this feature, such as MediaPlayer&rsquo;s Playlist window, Tracker&rsquo;s &ldquo;move to trash&rdquo;, and others.</p>
<p>jscipione fixed some background color management problems in <code>BTextView</code>.</p>
<p>nipos contributed a fix to <code>BControlLook</code> to use the &ldquo;panel text&rdquo; color for scrollbar arrows, fixing contrast in some color schemes. nephele came by later and fixed an oversight in the original change.</p>
<p>PulkoMandy added the remaining missing ASCII control character definitions to <code>InterfaceDefs.h</code>.</p>
<p>waddlesplash refactored <code>BObjectList&lt;&gt;</code> to take <code>bool owning</code> as a template parameter rather than a constructor parameter. This will make static code analysis of modules that use this class much easier (as there were a lot of &ldquo;double free/use-after-free&rdquo; false positives before that just had to be ignored), and also make memory leaks harder to inadvertently cause. He also cleaned up some code to use the more optimal <code>BStringList</code> instead of <code>BObjectList&lt;BString&gt;</code> at the same time.</p>
<p>nephele fixed <code>BFilePanel</code> to handle non-existent target directories more gracefully.</p>
<p>korli added checks to <code>BMenuField::SetLabel</code> to better handle <code>NULL</code> labels.</p>
<p>waddlesplash made some changes to <code>BList</code>, <code>BMessage</code>, and some of their consumers (like the registrar) to avoid allocating memory in more cases, and just use the stack (or defer it till later) instead.</p>
<p>waddlesplash adjusted and added some checks in <code>BScrollView</code> and <code>BColumnListView</code> to not redraw unnecessarily in more cases, improving performance in HaikuDepot somewhat.</p>
<p>ilzu fixed an error report in a corner-case the JSON message writer.</p>
<p>jscipione cleaned up the ownership of the &ldquo;menu semaphore&rdquo; in BWindow.</p>
<p>waddlesplash adjusted <code>BBlockCache</code> (a memory chunk cache) to not cache anything when a debug/guarded allocator is in use.</p>
<h3 id="servers">Servers</h3>
<p>korli added a hook to the VirtualKeyboard input panel (which is still disabled by default) to handle screen(mode) changes properly, and made it automatically refresh when the system keymap changes.</p>
<p>augiedoggie reworked screensavers to support using the &ldquo;system&rdquo; (i.e. UNIX user) password to unlock the screen after the screensaver run, and added a command-line argument to launch and lock the screen immediately instead of after a delay.</p>
<p>madmax fixed the name of a private method in <code>input_server</code>.</p>
<p>X512 removed an unnecessary argument from internal <code>draw_bezier</code> functions.</p>
<h3 id="drivers">Drivers</h3>
<p>Lt-Henry added support for AMD I2C busses to the <code>pch_i2c</code> driver.</p>
<p>waddlesplash reworked the &ldquo;IA32 BIOS&rdquo; bootloader to report the ACPI root pointer (same as the EFI bootloader), and to always use this in ACPI initialization rather than trying to re-find it.</p>
<p>waddlesplash fixed some corner-cases and added missing checks to XHCI initialization.</p>
<p>korli modified the <code>virtio_pci</code> bus driver to handle the device revisions more according to the specification.</p>
<p>waddlesplash adjusted the SCSI bus manager to use the proper maximum values for DMA attributes.</p>
<h3 id="file-systems">File systems</h3>
<p>waddlesplash optimized block fetching in B+tree iteration to not &ldquo;put&rdquo; and &ldquo;get&rdquo; the same block over and over again unnecessarily, speeding up &ldquo;git status&rdquo; performance with cold disk caches a bit (but there&rsquo;s still a lot more work to be done there.)</p>
<p>augiedoggie changed <code>userlandfs_server</code> to use <code>B_MULTIPLE_LAUNCH</code>, allowing multiple userlandfs/FUSE filesystems to be mounted at the same time.</p>
<p>waddlesplash added some missing locks to BFS&rsquo; checkfs operations, potentially fixing some rare KDLs.</p>
<h3 id="libroot--kernel">libroot &amp; kernel</h3>
<p>korli disabled &ldquo;C5&rdquo; and &ldquo;C6&rdquo; C-states on Intel &ldquo;Skylake&rdquo; microarchitecture, fixing the boot on some systems.</p>
<p>waddlesplash fixed the &ldquo;guarded heap object cache&rdquo; (a debug allocation facility) in the kernel after recent refactors.</p>
<p>waddlesplash added some sanity checks in <code>create_sem</code> to return errors when requesting a semaphore with a negative count, same as BeOS apparently did.</p>
<p>waddlesplash added checks in the file cache to validate that the size of the underlying file didn&rsquo;t shrink past the end of the request between when the read or write was issued and after actually acquiring the lock (and to bail out if it did). He also cleaned up some code related to page reservations.</p>
<p>korli adjusted runtime_loader to make <code>dlsym(RTLD_NEXT)</code> search all loaded ELF regions for the caller&rsquo;s address, not just the first ones. waddlesplash followed this up with some similar fixes to other parts of runtime_loader.</p>
<p>waddlesplash continued his ongoing memory management improvements, fixes, and cleanups, implementing more cases of resizing (expanding/shrinking) memory areas when there&rsquo;s a virtual memory reservation adjacent to them (and writing tests for these cases) in the kernel. These changes were the last remaining piece needed before the new <code>malloc</code> implementation for userland (mostly based on OpenBSD&rsquo;s malloc, but with a few additional optimizations and a Haiku-specific process-global cache added) could be merged and turned on by default. There were a number of followup fixes to the kernel and the new allocator&rsquo;s &ldquo;glue&rdquo; and global caching logic since, but the allocator has been in use in the nightlies for a few weeks with no serious issues. It provides modest performance improvements over the old allocator in most cases, and in some cases that were pathological for the old allocator (GCC LTO appears to have been one), provides order-of-magnitude (or mode) performance improvements.</p>
<p>korli fixed <code>pthread_key</code> destructor invocation to iterate over the keys for at least <code>PTHREAD_DESTRUCTOR_ITERATIONS</code>, as the specification indicates. waddlesplash made a small optimization to avoid syscalls in some cases of <code>pthread_cond_signal</code>.</p>
<p>waddlesplash added some more mutex acquisitions before spinlock acquisitions in the syslog code, avoiding some assertion failure panics when &ldquo;onscreen debug output&rdquo; is enabled for a boot.</p>
<p>waddlesplash refactored x86 APIC timer initialization, moving the calibration from the bootloader into the kernel, making it somewhat more accurate, and reading the value from CPUID on hypervisors rather than computing it from scratch.</p>
<p>waddlesplash added code to handle a corner-case in <code>mprotect</code>, fixing the last known kernel panic (assertion failure) that &ldquo;Iceweasel&rdquo; was regularly triggering on nightly builds.</p>
<p>waddlesplash fixed the kernel and bootloaders on x86 to not hang forever if the default serial line is dead or nonexistent. (On specialized or very new x86 hardware, like the Steam Deck, it doesn&rsquo;t exist, and trying to write to it without timeouts will just result in infinite hangs.)</p>
<p>phcoder contributed fixes to make <code>x86_{read|write}_msr</code> an inline function on 32-bit x86 (it already was on 64-bit), add X2APIC support to the EFI bootloader (needed on very recent hardware), use a separate allocation function to make sure that the &ldquo;PML4&rdquo; page is always allocated in the 32-bit region (which waddlesplash refactored and merged after memory management cleanups to other parts of the bootloader), and to support the GDT being above 4GB.</p>
<p>waddlesplash replaced some more implementations of standard C functions with versions from musl, including <code>memmove</code>, <code>strlen</code>, <code>strlcat</code>, and more. He also synchronized the <code>glob</code> implementation with upstream FreeBSD, added some more optimizations to <code>memcmp</code>, and more.</p>
<p>korli moved partial-zero-fill-pages logic from the file cache into VMCache, fixing <code>mmap</code> behavior in some corner cases involving RAMFS.</p>
<p>waddlesplash rewrote the kernel FIFO (<code>pipe(2)</code>) implementation to use more atomics and fewer locks, improving the performance of <code>stress-ng --pipe 1</code> (in a VM) from ~230 MB/s to ~2.5 GB/s.</p>
<h3 id="build-system">Build system</h3>
<p>korli fixed some miscellaneous warnings.</p>
<p>waddlesplash made a number of fixes to make it possible to build Haiku on FreeBSD, and Linux distributions that use musl. He also deleted some unneeded source files from the copy of <code>unzip</code> used as part of the build tools.</p>
<p>korli adjusted the stack protection options in the build system (which are disabled by default) to apply to more parts of the source tree.</p>
<p>nephele changed a usage of the <code>which</code> command to use the shell builtin <code>command -v</code> instead.</p>
<h3 id="documentation">Documentation</h3>
<p>waddlesplash dropped some obsolete internals documentation on merging patches from NetBSD trunk for <code>netresolv</code>, as the procedure described there doesn&rsquo;t apply anymore. Following some discussion on the mailing list, PulkoMandy wrote up a new page describing the general procedure for merging code from other sources into Haiku and keeping it up-to-date.</p>
<p>kuku929 (a new contributor!) contributed a graph showing an example of the kernel device manager&rsquo;s node tree.</p>
<p>PulkoMandy moved syscalls documentation from the website to the internals documentation, and imported oco&rsquo;s article on profiling applications.</p>
<p>dalmemail fixed a typo in a documentation comment for <code>VMCache</code>.</p>
<p>madmax added a missing &ldquo;end comment&rdquo; to the Haiku Book that was preventing the &ldquo;Synchronization Primitives&rdquo; page from being rendered.</p>
<h3 id="thats-all-folks">That&rsquo;s all, folks!</h3>
<p>Thanks again to all who contribute to Haiku, and especially those donors who make my contract possible!</p></description></item><item><title>Haiku Activity & Contract Report, January 2025</title><link>https://www.haiku-os.org/blog/waddlesplash/2025-02-10-haiku_activity_contract_report_january_2025/</link><author>waddlesplash</author><pubDate>Mon, 10 Feb 2025 23:20:00 -0500</pubDate><guid>https://www.haiku-os.org/blog/waddlesplash/2025-02-10-haiku_activity_contract_report_january_2025/</guid><description><p>This report covers hrev58487 through hrev58583.</p>
<h3 id="applications">Applications</h3>
<p>omesh-barhate contributed a patch to add a context menu to WebPositive&rsquo;s bookmark bar items, to allow them to be edited more easily. humdinger cleaned up the strings in the new context menu and enhanced its functionality, and removed the &ldquo;move to front&rdquo; logic in the Downloads window.</p>
<p>digitalbox98 (a new contributor!) contributed a change to add a &ldquo;Remove&rdquo; menu for Transformers in Icon-O-Matic. Zardshard cleaned up some variable names in Icon-O-Matic as well.</p>
<p>PawanYr (a new contributor!) made TeamMonitor group child teams under their parents. (This makes multiprocess web browsers and terminals easier to deal with.)</p>
<p>PawanYr made Backgrounds preferences update the view whenever the X/Y coordinate boxes are modifed.</p>
<p>PawanYr fixed a crash related to query templates in Tracker. jscipione adjusted Tracker to disallow control characters in the file name edit box. nephele fixed the button color in file panels.</p>
<p>jscipione cleaned up and added more invocations of <code>AdoptSystemColors</code> and the like in multiple applications and classes: Deskbar, Tracker, HaikuDepot, BTextView, and more.</p>
<p>nipos made the analog clock display in Time use the &ldquo;document&rdquo; color rather than a hard-coded white color.</p>
<p>jscipione completed and merged a very large refactor of menu items and shortcuts handling in Tracker. This touches nearly every item displayed in menu bars and context menus in Tracker, and implements live updating of menus when modifier keys change, adds Cut/Copy/Paste to context menus, and more.</p>
<p>nipos added an option to strikeout text in StyledEdit.</p>
<p>PawanYr added speed, acceleration, and scroll settings to Touchpad preferences.</p>
<p>nephele deleted a bunch of obsolete and unused code from Appearance preferences, and renamed some classes to match Haiku conventions.</p>
<h3 id="command-line-tools">Command line tools</h3>
<p>augiedoggie fixed <code>rc -d</code> to correctly decompile <code>app_version</code> resources that were built on Haiku as well as on BeOS.</p>
<p>catmeow72 (a new contributor!) added a <code>-B</code> / <code>--get-brightness</code> option to <code>screenmode</code> to retrive the current brightness (instead of just setting it.)</p>
<h3 id="kits">Kits</h3>
<p>waddlesplash moved the system colormap to a shared area, avoiding all GUI applications needing to fetch and store it in their own memory.</p>
<p>nipos fixed tooltips in &ldquo;channel slider&rdquo; controls to display the expected values (rather than the &ldquo;raw&rdquo; values, which were 1000x larger.)</p>
<p>jscipione made BColorControl call <code>SetValue()</code> and <code>Invoke()</code> on color drops, if the control was enabled.</p>
<p>jscipione moved the ColorPreview classes to the private &ldquo;shared&rdquo; kit, deleting duplicate code between Appearance preferences and Terminal.</p>
<p>wadlesplash cleaned up the handling of ELF segments in the Debug Kit, fixing a lot of logic that did not handle multiple TEXT segments correctly (and Clang/LLD generate binaries with multiple TEXT segments by default, while GCC/LD can be configured to do so.) He also restored support for lookup of symbols based on remote memory access (instead of only ELF files), among other things. This fixes the generation of stack traces in debug reports for applications compiled with Clang (including most of the web browser ports.)</p>
<p>daveslusher (a new contributor!) fixed some potential buffer truncation problems when looking up MIME types.</p>
<h3 id="servers">Servers</h3>
<p>X512 cleaned up some error handling code in app_server.</p>
<p>nipos implemented <code>B_STRIKEOUT_FACE</code> in app_server.</p>
<h3 id="drivers">Drivers</h3>
<p>waddlesplash improved the device probing logic in the FreeBSD compatibility layer to not perform so many string comparisons, and to avoid allocating and freeing big chunks of memory so often.</p>
<p>Lt-Henry adjusted the HID layer to parse &ldquo;feature&rdquo; reports (as well as &ldquo;input&rdquo; reports.)</p>
<p>kallisti5 adjusted radeon_hd to work properly after area cloning changes in the kernel (detailed below.)</p>
<p>DruSatori (a new contributor!) contributed support for &ldquo;Alder Lake&rdquo; chipsets to the <code>intel_extreme</code> modesetting driver.</p>
<p>korli added an <code>amd_thermal</code> driver to fetch temperatures on AMD systems, cleaned up the <code>pch_thermal</code> driver, and added both of those as well as the <code>acpi_thermal</code> driver to the default builds.</p>
<p>ilzu contributed support for the CTH-470 to the Wacom driver.</p>
<h3 id="file-systems">File systems</h3>
<p>waddlesplash added a missing check for parent directores in packagefs, fixing a rare KDL.</p>
<p>Jim906 contributed changes to make the NFSv4 driver build under the &ldquo;userlandfs&rdquo; infrastructure, which should make it much easier to debug. He also fixed some port buffer sizing problems in the userlandfs layer itself.</p>
<p>waddlesplash added support for &ldquo;special&rdquo; files (like FIFOs or UNIX sockets) to RAMFS. He also cleaned up some duplicate code, reduced lock contention, made the <code>readdir()</code> hook return more than one <code>dirent</code> at a time, and fixed some other bugs as well.</p>
<h3 id="libroot--kernel">libroot &amp; kernel</h3>
<p>waddlesplash&rsquo;s work on memory management logic in the kernel (and elsewhere) continued in January, with more fixes and improvements to page mapping, memory reservation accounting, kernel malloc block sizes, kernel stack allocation, cache fault counting, and much more. He also got OpenBSD&rsquo;s malloc up to performance par with libroot&rsquo;s current malloc (or better, in many cases) even under multithreaded workloads, though there&rsquo;s still more that needs to be done before it can be merged.</p>
<p>waddlesplash made some changes to avoid holding the &ldquo;hot vnodes&rdquo; lock too long in the VFS low-memory handler, fixing deadlocks.</p>
<p>waddlesplash adjusted the FD table creation logic in the kernel, shrinking the initial table size and breaking up some large allocations to make it much more likely to hit fast-paths.</p>
<p>cmeerw submitted a patch to libroot to fix an infinite loop in <code>pipe2</code> with Emacs/GNUlib.</p>
<p>waddlesplash added a fast path in the kernel thread code for when a thread is fetching its own <code>Thread</code> structure.</p>
<p>waddlesplash merged the bootloader and runtime_loader heap (malloc) implementations, as they were mostly copies of one another with differences in glue code only. He also fixed and implemented some tests for this heap, improved the glue code in runtime_loader, and did a bit of optimization of memory usage in the bootloader while at it.</p>
<p>waddlesplash fixed some mishandling of <code>errno</code> in the filesystem block cache that was leading to KDLs in some circumstances.</p>
<p>waddlesplash added an error definition for <code>ESOCKTNOSUPPORT</code>, which is in POSIX-2024.</p>
<p>waddlesplash adjusted permissions checks in the kernel to block teams from cloning areas they don&rsquo;t own with more permissions than the source area has (write permissions, specifically), and also to block non-root teams from inspecting other teams&rsquo; memory properties.</p>
<p>waddlesplash moved the default selection of <code>B_RANDOMIZED_*</code> area types (i.e. ASLR) to libroot, so that it can be overridden for individual areas (if necessary).</p>
<p>waddlesplash added a <code>CLONE_AREA</code> message to the userspace debugger support in the kernel (used by the Debug Kit fixes described above), and added some more permissions checks to the user_debugger logic in general.</p>
<p>waddlesplash improved a permissions check in the <code>area_for</code> implementation, and made it distinguish between regular and &ldquo;reserved&rdquo; areas.</p>
<p>waddlesplash refactored the kernel&rsquo;s userspace symbol lookup logic to rely on the <code>extended_image_info</code> structures (which the <code>runtime_loader</code> reports to the kernel) rather than cloning the runtime_loader&rsquo;s debug information area. He also switched the <code>image_info</code> structures to use the standard <code>DoublyLinkedList</code> class, and cleaned up some other code related to linked-lists as well.</p>
<p>waddlesplash cleaned up code in runtime_loader, and then made it properly respect the <code>PF_EXECUTE</code> program header flag. Previously it just made all non-writable regions executable (which matched what GCC/LD generally generated by default anyway, but Clang/LLD binaries actually do have a non-executable TEXT section in most cases.) He also cleaned up the kernel ELF loader to not set user protections (based on <code>PF_*</code> flags) until the very end.</p>
<p>waddlesplash cleaned up a redundant <code>wait_for_thread</code> syscall.</p>
<p>waddlesplash improved the address-space blocking for uninitialized/freed memory. Under <code>PARANOID_KERNEL_MALLOC</code> (which is enabled for nightly builds), this reserves the virtual addresses most likely to be hit if &ldquo;uninitialized&rdquo; (<code>0xcccccccc</code>) or freed (<code>0xdeadbeef</code>) memory is touched. It only tried to reserve the 32-bit versions of the addresses, which wasn&rsquo;t very useful on 64-bit systems; now it reserves the 64-bit addresses as well.</p>
<p>waddlesplash changed page fault behavior to always insert new (clean) pages into the <em>topmost</em> cache object on page faults, rather than the lowest (deepest) cache object. In the case of a read fault, they would in some circumstances be inserted into the deepest cache instead. This behavior dated all the way back to NewOS, but now that Haiku has rather advanced copy-on-write, fork(), and cache resizing support, it was conflicting with other features (and probably isn&rsquo;t an optimization anyway as it would just cause more copy-on-writes when writes did occur.) This problem manifested in an extremely race-sensitive KDL, and after tracking it down and fixing it, waddlesplash added a new ASSERT that would have caught the problem immediately and without waiting for any races (and which has caught some other related problems already).</p>
<p>waddlesplash made the &ldquo;I/O context root&rdquo; lock (which must be held when <code>chroot</code>ing, among other things) an R/W lock, since the vast majority of the time an application&rsquo;s root path only needs to be read, not set, reducing lock contention.</p>
<p>waddlesplash reintroduced an &ldquo;aligned pointers&rdquo; optimization for <code>strcmp</code>, reducing CPU time/usage especially for longer strings as up to 4 bytes can be compared at a time now. He also replaced some other string functions with their musl equivalents.</p>
<h3 id="arm--risc-v">ARM &amp; RISC-V</h3>
<p>kallisti5 fixed some logic in the bootloader and serial code to fail properly if clock frequencies can&rsquo;t be fetched or are invalid.</p>
<h3 id="build-system">Build system</h3>
<p>kallisti5 made some various fixes and improved support for architectures other than x86_64 in the build-cross-compiler Docker scripts.</p>
<p>waddlesplash fixed some missing build dependency declarations in the packagefs rules.</p>
<p>waddlesplash deleted some no-longer-needed license text files and cleaned up some of the remaining ones.</p>
<p>waddlesplash improved some error messages around Python detection in <code>./configure</code>.</p>
<p>daveslusher fixed some regular expression errors in Awk code that processes USB and PCI device lists.</p>
<h3 id="documentation">Documentation</h3>
<p>nipos deleted some old and unused configuration files from the documentation directories, and replaced the remaining references to &ldquo;OpenBeOS&rdquo; with &ldquo;Haiku&rdquo;.</p>
<p>jscipione reordered the keys enum in InterfaceDefs, and PulkoMandy added the missing elements (ASCII control codes and characters) for clarity and completeness.</p>
<h3 id="thats-all-folks">That&rsquo;s all, folks!</h3>
<p>Thanks again to all who contribute to Haiku, and especially those donors who make my contract possible!</p></description></item><item><title>Haiku, Inc. Financial Report for 2024 is now available</title><link>https://www.haiku-os.org/news/2025-02-01_2024_financial_report/</link><author>leavengood</author><pubDate>Sat, 01 Feb 2025 11:00:00 -0500</pubDate><guid>https://www.haiku-os.org/news/2025-02-01_2024_financial_report/</guid><description><p>The Haiku, Inc. <a href="https://www.haiku-inc.org/docs/haiku_inc-financial-report-2024.pdf">financial report for 2024</a>
is now available on the <a href="https://www.haiku-inc.org/documents/">Haiku, Inc. Documents page</a>.</p>
<p>Our donations for 2024 were higher than any other year, breaking all previous
records by almost $10,000! Even with some reasonably high expenses we still
managed to have a net positive year.</p>
<p>In 2024 our contractor waddlesplash worked the whole year, and we had a new beta
release.</p>
<p>We had some very generous large donations this year, which is part of what made
this a record setting year. One individual in particular donated a total of
$7,500 through GitHub Sponsors, which is extremely generous!</p>
<p>We appreciate every donation, big and small, and every donor plays a role in
helping to keep Haiku moving forward, so again: <strong>thank you so much!</strong></p>
<p>Bitcoin is at record highs, and so therefore is our Coinbase balance. Though as
noted in the report we still have to sort out details of our Coinbase account to
make those funds more accessible.</p>
<p>Finally, while this is a news item about the financial report, I want to extend
a thanks to all our volunteers as well. While our donors play their part, so do
our volunteers, continuing to improve Haiku on their own time, year after year.
Not just developers, we also have volunteers who help with documentation,
answering questions in the forum, and working on translations. Plus maybe that
person who manages the finances and works on the financial reports&hellip;</p>
<p>Thanks to all who help keep Haiku moving forward!</p></description></item><item><title>Haiku Activity & Contract Report, December 2024 (ft. Iceweasel)</title><link>https://www.haiku-os.org/blog/waddlesplash/2025-01-13-haiku_activity_contract_report_december_2024/</link><author>waddlesplash</author><pubDate>Mon, 13 Jan 2025 23:30:00 -0500</pubDate><guid>https://www.haiku-os.org/blog/waddlesplash/2025-01-13-haiku_activity_contract_report_december_2024/</guid><description><p>This report covers hrev58369 through hrev58486.</p>
<h2 id="iceweasel">Iceweasel</h2>
<p>The biggest piece of news from last month is the arrival of &ldquo;Iceweasel&rdquo;, a web browser built from Mozilla Firefox source code but without any official branding or registered trademarks, in the software depots (for x86_64 only, at the moment.) We&rsquo;ve been &ldquo;slow-rolling&rdquo; the announcements on this one, in part because the browser was quite unstable at first and prone to cause kernel assertion failures on the nightly builds, but after a month of work it&rsquo;s in much better shape and is relatively stable.</p>
<p>There are still some kernel assertion failures that happen intermittently on the nightly builds with it (but not on beta5, as assertions are disabled there; you may get suboptimal performance or over-reservations of memory by the web browser processes, but the problems causing the failed assertions shouldn&rsquo;t be any more serious than that), and there are a lot of issues or missing features with the port itself still to resolve (e.g. WebRTC isn&rsquo;t implemented at all, pop-up windows/menus from browser extensions may not accept any mouse input, etc.), but these and other issues are slowly being worked on.</p>
<h2 id="memory-management-changes">Memory management changes</h2>
<p>Motivated in large part by Iceweasel, I (waddlesplash) spent a significant amount of time in December (and still ongoing, in fact) working on various parts of the kernel virtual memory management facilities. Many of these changes were aimed at fixing problems exposed or suggested by investigating matters in Iceweasel and other memory-management-intensive tests, such as the Boehm garbage collector.</p>
<p>The changes in this area are probably too many (dozens) and too technical to reasonably detail all of them here (so check the commit logs if you&rsquo;re particularly interested), but among these are: optimizations to batch page writes in some cases, more safety checks in the area resize logic, removal of obsolete fields and comments, many fixes to memory reservation (commitment) behavior (especially in areas with per-page protections), more testcases in the test suite for corner-cases, better handling of reporting initially available memory, fixing of some softlocks that could occur on low-memory conditions, fixes to swap support, enforcement of more protection restrictions, major consolidation of page unmapping code from per-architecture files into architecture-independent files, and many more.</p>
<h3 id="applications">Applications</h3>
<p>nipos contributed a fix to Backgrounds to ignore color drops if the color picker is disabled.</p>
<p>apl did some significant refactoring to HaikuDepot&rsquo;s data models to pave the way for future improvements. He also adjusted the logic around package installation to report a &ldquo;pending&rdquo; state more rapidly, and hid the ratings UI when a package can&rsquo;t be rated for any reason.</p>
<p>PulkoMandy enhanced the Cortex &ldquo;LoggingConsumer&rdquo;, a media node that logs details about whatever media buffers were sent to it, to report more details.</p>
<p>jscipione added back spacing at the bottom of the Expander main window. He also made some improvements to the font selection view in Appearance preferences, and nephele deleted some dead code from them as well.</p>
<p>jscipione made a number of changes to Tracker to fix &ldquo;Escape&rdquo; key behavior in file panels when &ldquo;typeahead filtering&rdquo; is enabled, and cleaned up the display of &ldquo;selections&rdquo; in the Find window. He made a number of coding style cleanups as well.</p>
<p>waddlesplash made a number of cleanups to the battery display in PowerStatus, making its display more intuitive and easier to tell at a glance what&rsquo;s going on when looking at it in Deskbar.</p>
<p>nephele made a number of improvements to the error log window in <code>mail_daemon</code>, such as support for dark mode and better scrollbar proportions.</p>
<p>jackburton79 fixed a potential memory leak on error and added some sane initialization defaults to Terminal.</p>
<p>humdinger made Sounds preferences remember the file panel location, and show only files and not folders in menus. He also added a mechanism to preview sound files in the file panel, and cleaned up a few of the names of notification sounds.</p>
<h3 id="command-line-tools">Command line tools</h3>
<p>waddlesplash (with some help from madmax) implemented <code>pkgman search -i --not-required</code>, a mode that shows only installed packages that are not required by any other installed package (i.e. &ldquo;leaf&rdquo; packages.)</p>
<p>kallisti5 adjusted the <code>route</code> command to accept an empty network address to mean &ldquo;default&rdquo;, which OpenVPN uses to set up the default route.</p>
<h3 id="kits">Kits</h3>
<p>waddlesplash adjusted BTimedEventQueue (used for all audio output and other media operations on Haiku) to properly flush events on destruction, and a few other minor improvements.</p>
<p>waddlesplash adjusted the naming of the new <code>input_pointing_device_subtype</code> enum, which is used to report what kind of device generated pointer input events.</p>
<p>PulkoMandy implemented custom baudrates for serial output from the POSIX APIs all the way down into the USB serial driver.</p>
<p>nipos improved the legibility of text labels in the media controls display.</p>
<h3 id="servers">Servers</h3>
<p>X512 contributed changes to drop the legacy &ldquo;2D acceleration&rdquo; and batched-buffer-copies logic from app_server. This was not enabled (or even compiled in) for many years; it&rsquo;s obsolete even for the old hardware that supported these features (as CPU routines to do the same are faster much of the time anyway, it seems), and &ldquo;modern&rdquo; 2D-accelerated graphics just go through the 3D pipelines, so if app_server ever gained such code it would share practically nothing with these anyway.</p>
<p>jscipione redid the notification panel sizing to be properly font-size-aware. He adjusted the notification icons sizing in various servers as well.</p>
<h3 id="drivers">Drivers</h3>
<p>korli fixed some problems with the handling of queue numbers in the <code>virtio-pci</code> bus driver, fixing a number of intermittent issues with virtio devices (they would sometimes fail to initialize, among other problems.)</p>
<p>waddlesplash sychronized the <code>realtekwifi</code> driver and WiFi stack with upstream FreeBSD, bringing in a number of bugfixes and other enhancements. He also tweaked the compiler flags to drop more unused or unneeded code at compile time.</p>
<p>waddlesplash restored the use of &ldquo;uncached&rdquo; memory for the ACPI module on x86(_64) by default, fixing boot failures on a few devices.</p>
<p>mtl1979 contributed checks to the usb_disk driver to return errors immediately rather than trying to read or write if there&rsquo;s a status of &ldquo;no media present&rdquo;.</p>
<p>Lt-Henry added a new device ID to the Intel modesetting graphics driver.</p>
<p>PulkoMandy fixed some compiler warnings and enabled <code>-Werror</code> in various drivers. He also added more error checking to the PS/2 driver, to allow long timeouts to be avoided in another case.</p>
<p>waddlesplash fixed a race condition in the SCSI bus manager&rsquo;s (relatively new) request notification logic, and added some flags to make sure that we don&rsquo;t wait for memory in the critical path.</p>
<p>PulkoMandy enabled VESA BIOS patching (to inject more video modes) by default on hardware where it is known to work.</p>
<p>waddlesplash fixed a memory leak in the network stack when &ldquo;raw&rdquo; sockets were closed, and added an <code>ASSERT</code> that would have caught the problem.</p>
<h3 id="file-systems">File systems</h3>
<p>Jim906 contributed a fix to the FAT filesystem&rsquo;s entry-caching logic: previously, multiple instances of the same filename with different casing could wind up in the entry cache when the file was not present, but these weren&rsquo;t properly cleared when a file with one of those cases was created. Now, such incorrect &ldquo;missing&rdquo; entries aren&rsquo;t created at all, fixing some corner-cases around file copying in Tracker and allowing some FAT-specific code to be deleted.</p>
<p>waddlesplash moved handling of <code>open(O_RDONLY | O_TRUNC)</code> from filesystems into the VFS, and changed its behavior to return an error rather than truncating the file. What these flags specified together do is &ldquo;undefined&rdquo; in POSIX, but the Linux manpages ominously state &ldquo;On many systems the file is actually truncated&rdquo;, and indeed on Linux, using this open mode actually truncates files most of the time! That&rsquo;s a rather surprising result, and there doesn&rsquo;t seem to be much reason to support it, so Haiku now does the saner thing and refuses to do anything in this case.</p>
<p>waddlesplash adjusted the filesystem query parser to not try to read the key size of invalid indexes in a corner case.</p>
<p>waddlesplash adjusted the filesystems logging during disk scanning and mounting to only print errors when a disk at least partially looks like a particular filesystem but is otherwise invalid, rather than printing a lot of noisy error messages for every failed identification. He also cleaned up some code related to initializing filesystems, and fixed the logic around initializing NTFS disks.</p>
<p>Jim906 fixed <code>file_cache_read</code> under userlandfs; previously it would return errors when reaching the end of files rather than handling them correctly.</p>
<p>waddlesplash added allocation of &ldquo;scratch&rdquo; buffers to the packagefs memory cache systems, avoiding frequent allocation and deallocation of temporary buffers.</p>
<h3 id="libroot--kernel">libroot &amp; kernel</h3>
<p>waddlesplash added a missing call to a base method in the <code>FileDiskDevice</code> class (used for handling files registered as virtual disks), fixing a file descriptor leak. He also cleaned up the code style a bit and fixed some minor TODOs while at it.</p>
<p>waddlesplash cleaned up some code in the &ldquo;synchronous I/O&rdquo; routines in the kernel (which are used when asynchronous I/O isn&rsquo;t possible at the lowest level), and fixed an assertion check that wasn&rsquo;t accounting for all possible error conditions, fixing some assertions when using files as disk devices.</p>
<p>waddlesplash fixed a bug in the logic that handled opening symlinks to nonexistent files with <code>O_CREAT</code> in the VFS. He also added a testcase for this and another related issue.</p>
<p>waddlesplash fixed some minor bugs in the &ldquo;scheduling recorder&rdquo; (kernel subsystem that records when threads are scheduled/descheduled, for performance analysis) and &ldquo;DebugAnalyzer&rdquo; (the GUI app that analyzes such reports).</p>
<p>waddlesplash fixed an unlocking-order problem in a VFS low-resource hook that occasionally caused kernel panics on high memory usage.</p>
<p>waddlesplash dropped the <code>*rand48_r</code> methods from <code>stdlib.h</code>. They&rsquo;re kept in <code>libroot</code> for ABI compatibility, but as they&rsquo;re nonstandard extensions provided only by glibc (not even the BSDs have them), we don&rsquo;t need to export them in public headers anymore.</p>
<p>waddlesplash renamed the <code>MoveFrom</code> primitive in the kernel <code>LinkedList</code> to be called <code>TakeFrom</code> instead. It doesn&rsquo;t have &ldquo;move semantics&rdquo;, but rather appends the contents of the other list onto the current one, so we should avoid using the word &ldquo;move&rdquo; in such cases to avoid confusion.</p>
<p>waddlesplash adjusted <code>vfork()</code> to not call any fork hooks. (We don&rsquo;t implement a &ldquo;true&rdquo; <code>vfork</code>, but we can at least avoid doing most of the things that <code>fork</code> does in it, and we now do.)</p>
<p>korli implemented <code>pthread_getcpuclockid</code>, which retrieves a CPU time clock ID for a specific thread rather than a whole process.</p>
<p>waddlesplash enabled more <code>printf</code>-related warnings for bootloader code.</p>
<p>Anarchos contributed a change to increase the size of the memory region used for decompressing the boot tar archive, which is used when booting from certain types of media (like network devices).</p>
<h3 id="build-system">Build system</h3>
<p>waddlesplash added some sanity checks to the core build rules to prevent setting of <code>ConfigVar</code>s with no value, and also to remove the <code>@profile</code> from <code>JAM_TARGETS</code> to avoid spurious warnings/errors later on.</p>
<p>kallisti5 added handling for ARM and RISC-V <code>HOST_CPU</code>s in the core build rules.</p>
<h3 id="documentation">Documentation</h3>
<p>OscarL fixed a typo in the &ldquo;Introduction to the <code>launch_daemon</code>&rdquo; documentation.</p>
<p>jscipione added documentation for <code>BTextView::{GetFontAndColor,SetFontAndColor}</code>.</p>
<p>cafeina contributed documentation for <code>BShelf</code> and <code>BGameSound</code>.</p>
<h3 id="thats-all-folks">That&rsquo;s all, folks!</h3>
<p>Thanks again to all who contribute to Haiku, and especially those donors who make my contract possible! Haiku, Inc. collected around $30,000 of donations in 2024; if we get even just that much this year, I&rsquo;ll be able to spend more paid time on Haiku without exhausting the Inc.&rsquo;s funds too quickly.</p></description></item><item><title>Haiku Activity & Contract Report, November 2024</title><link>https://www.haiku-os.org/blog/waddlesplash/2024-12-10-haiku_activity_contract_report_november_2024/</link><author>waddlesplash</author><pubDate>Tue, 10 Dec 2024 23:45:00 -0500</pubDate><guid>https://www.haiku-os.org/blog/waddlesplash/2024-12-10-haiku_activity_contract_report_november_2024/</guid><description><p>This report covers hrev58292 through hrev58368.</p>
<h3 id="applications">Applications</h3>
<p>apl added an &ldquo;installed&rdquo; indicator to icons in HaikuDepot&rsquo;s list view, and continued the refactoring of its internal data model. humdinger contributed icons for the &ldquo;native&rdquo; indicator, and apl added this to HaikuDepot&rsquo;s display as well.</p>
<p>korli fixed an occasional crash in Screen preferences.</p>
<h3 id="kits">Kits</h3>
<p>X512 fixed an incorrect port deletion in <code>BApplication</code>.</p>
<p>X512 added move constructor and assignment support to <code>BRegion</code> and <code>BShape</code>, and added a 2-argument constructor variant to BRect.</p>
<p>waddlesplash restored some special casing for <code>B_RGB32</code> data in <code>BBitmap::SetBits</code>, fixing some compatibility with BeOS, and korli adjusted it a bit for correctness.</p>
<p>linkmauve fixed a typo in an assertion in <code>BTimeSource</code>.</p>
<p>waddlesplash completely rewrote <code>BTimedEventQueue</code>. The new implementation avoids <code>malloc</code> (necessary as this is in the &ldquo;hot path&rdquo; for media playback) and uses a templated linked-list rather than a manual one. Multiple users have reported that media playback seems smoother, especially on lower-end hardware, after this change.</p>
<h3 id="servers">Servers</h3>
<p>Hanicef (a new contributor!) contributed fixes to PadBlocker (the input_server filter that handles blocking touchpad events when keyboards are in use) to handle key repeat properly, and also to avoid blocking events from non-touchpad devices at all.</p>
<p>waddlesplash adjusted net_server to always remove the &ldquo;autoconfiguration client&rdquo; (usually the DHCP client) when a network link goes inactive, rather than just retry forever when there&rsquo;s no way to send data at all.</p>
<h3 id="drivers">Drivers</h3>
<p>waddlesplash fixed some old bugs in the SCSI layer where flags were reset on the wrong request, which was the cause of many &ldquo;no such range!&rdquo; KDLs. He then removed a number of unneeded fields from internal data structures, and removed the need for the <code>locked_pool</code> memory allocation module, saving some memory and slightly increasing efficiency. The SCSI driver was the only consumer of the <code>locked_pool</code> system, which is now totally obsoleted by the kernel&rsquo;s own memory management routines, so it was deleted entirely.</p>
<p>korli fixed some memory management code in the es1370 and auvia drivers, fixing crashes and audio output on some systems that use them.</p>
<p>waddlesplash refactored the network stack to return &ldquo;ancillary data&rdquo; (mostly relevant for UNIX domain sockets, as this allows file descriptors to b shared across processes) all in one chunk if possible, rather than separate chunks. The POSIX specification doesn&rsquo;t seem to require this, but Firefox&rsquo;s IPC code (which was taken from Chromium) seems to expect it.</p>
<p>waddlesplash made some adjustments to the network stack and the FreeBSD compatibility layer to report traffic statistics much more reliably (and to report driver-level statistics at all.)</p>
<p>waddlesplash added a missing cancellation of queued transfers to the USB disk driver, fixing KDLs on failures to attach to USB floppy drives.</p>
<p>waddlesplash added another USB device ID to the <code>realtekwifi</code> driver, and later synchronized it (and the net80211 stack) with FreeBSD. He also adjusted some routines in the compatibility layer for correctness, allowing a Haiku-specific patch to be dropped from the stack.</p>
<p>waddlesplash added a missing NULL check and some missing deletions in the USB stack, fixing some (long-standing) memory leaks and (more recent) assertion failures.</p>
<p>PulkoMandy reworked the (incomplete) USB webcam driver to use standard headers rather than redefining constants from the UVC specification itself, and added support for decoding more information from UVC descriptors to <code>listusb</code>.</p>
<p>PulkoMandy fixed handling of unplugs and canceled transfers in the RNDIS network driver (used for USB tethering.)</p>
<p>waddlesplash (following analysis by PulkoMandy) increased the default timeout used in PS/2 mouse resets, fixing initialization on some hardware.</p>
<h3 id="file-systems">File systems</h3>
<p>korli restored the FAT driver reporting volume names with lowercase letters (the behavior from before the driver was refactored.)</p>
<p>waddlesplash fixed some memory assertion failures in RAMFS and ram_disk, and did some minor cleanups as well.</p>
<p>Jim906 implemented prefetching in the block_cache, and adjusted the FAT driver to make use of it, greatly speeding up mounts of larger FAT partitions.</p>
<p>dalme contributed a patch to clarify the meaning of some fields in BFS initialization.</p>
<p>waddlesplash fixed BFS&rsquo;s asynchronous I/O routine to keep references to inodes properly (and also fixed similar issues in the FAT and EXFAT drivers as well), fixing a very old assertion failure ticket.</p>
<p>waddlesplash fixed handling of the &ldquo;last modified&rdquo; index in BFS queries, and adjusted type coercion in the query parser to behave correctly for &ldquo;time&rdquo; types (and avoid re-converting types when unneeded.)</p>
<p>waddlesplash reworked how query terms are &ldquo;scored&rdquo; (in order to pick what index to run the query off of), signficantly improving efficiency by picking smaller indexes in many cases.</p>
<p>OscarL and waddlesplash reworked how partitions are matched in mount_server, reducing the number of &ldquo;false positives&rdquo; and partitions automounted erroneously.</p>
<h3 id="libroot--kernel">libroot &amp; kernel</h3>
<p>waddlesplash fixed some corner-cases in handling of FD arrays for <code>select</code>, and moved some definitions from <code>sys/select.h</code> to private headers.</p>
<p>korli added a number of functions specified in POSIX-2024, mostly to open FDs with <code>O_CLOEXEC</code> set automatically, such as <code>pipe3</code>, <code>dup2</code>, and memory management routines, like <code>reallocarray</code>, <code>memmem</code>, and some miscellaneous methods besides.</p>
<p>waddlesplash fixed an issue in the kernel <code>object_cache</code> arena allocation system where the &ldquo;minimum object reserve&rdquo; setting wasn&rsquo;t always respected properly.</p>
<p>waddlesplash added code to the bootloader to display the size of the bootable partitions in the menus, if that information can be retrieved.</p>
<p>waddlesplash added a necessary x86-specific stub to the libroot stubs files, which are used when &ldquo;bootstrapping&rdquo; (building a system fully from source with no built dependencies.)</p>
<p>waddlesplash did some cleanup to code in libroot, making one function return <code>PTHREAD_CANCELED</code> as the spec says it should, and using more common methods for wait time conversion where possible.</p>
<p>X512 added <code>posix_devctl</code> to libroot, a new POSIX-2024 method intended to replace the variable-argument <code>ioctl</code>. Meanwhile, waddlesplash refactored the <code>ioctl</code> declaration to use macros rather than struct initializers to avoid using the variable-argument <code>ioctl</code> when possible.</p>
<p>waddlesplash unified the bitmap blitting routines and image placement computations used for displaying the startup splash between the bootloader and the kernel.</p>
<p>korli added support for RGB16 framebuffers in the EFI loader.</p>
<p>waddlesplash refactored the &ldquo;unused vnodes&rdquo; system in the kernel VFS layer to use an inline template linked-list, and a spinlock to protect it rather than a (heavily contended, during busy FS operations like <code>git status</code>) mutex, which requires use of linked-lists anyway. He also adjusted the <code>io_context</code> structure to be R/W-locked, and added a fast path out of the vnode lookup routine. Put together, these changes significantly improve <code>git status</code> performance, at least when there&rsquo;s a hot disk cache.</p>
<p>waddlesplash refactored and cleaned up some parts of the <code>do_iterative_fd_io</code> and <code>do_fd_io</code> routines in the kernel VFS layer (which are used in filesystems to implement kernel-level asynchronous and vectored I/O), to fix some corner cases and handle references more correctly. This was then used to clean up the block_cache&rsquo;s new prefetcher a bit more.</p>
<p>waddlesplash fixed the default buffer size in <code>KPath</code>, the kernel path buffer handling class.</p>
<p>waddlesplash added a mechanism for the &ldquo;framebuffer console&rdquo;, used to display KDLs, to be initialized much earlier in the kernel startup process, allowing kernel panics from virtual memory initialization (among other things) to be displayed onscreen, not just written to the syslog.</p>
<p>waddlesplash cleaned up and added more commands for dealing with I/O schedulers in the kernel debugger.</p>
<p>waddlesplash made a number of fixes to the virtual memory system&rsquo;s handling of <code>mprotect</code> and related functions, fixing assertion failures and other issues when these are used in certain circumstances, especially in conjunction with <code>fork</code>. He also turned down the logging of a variety of the page fault methods.</p>
<p>PulkoMandy added mkostemp (specified in POSIX-2024.)</p>
<h3 id="documentation">Documentation</h3>
<p>PulkoMandy added documentation for some of the implementation details of <code>ioctl</code> and <code>posix_devctl</code> on Haiku.</p>
<h3 id="thats-all-folks">That&rsquo;s all, folks!</h3>
<p>Thanks again to all who contribute to Haiku, and especially those donors who make my contract possible.</p></description></item><item><title>Haiku Activity & Contract Report, October 2024</title><link>https://www.haiku-os.org/blog/waddlesplash/2024-11-06-haiku_activity_contract_report_october_2024/</link><author>waddlesplash</author><pubDate>Tue, 05 Nov 2024 17:30:00 -0500</pubDate><guid>https://www.haiku-os.org/blog/waddlesplash/2024-11-06-haiku_activity_contract_report_october_2024/</guid><description><p>This report covers hrev58188 through hrev58291.</p>
<h3 id="applications">Applications</h3>
<p>nipos adjusted Icon-O-Matic to open file panels in the same folder as the current file, fixed a bug in FontDemo that was preventing direct selection of font families, fixed Terminal to not change view sizes when changing font sizes, fixed text colors in Screen and ScreenSaver preferences, and more.</p>
<p>apl fixed the &ldquo;Open&rdquo; button in HaikuDepot not displaying application names in some circumstances, refactored HaikuDepot&rsquo;s user ratings code, improved default the logic around default icons, and more.</p>
<p>waddlesplash added <code>MoveOnScreen()</code> calls to a number of applications and preferences, to avoid having windows start offscreen after screen sizes changed. pinaraf contributed a similar fix for the Printers preflet.</p>
<p>sen (a new contributor!) made a number of improvements to the PowerStatus application to better support systems with multiple batteries, including proper handling for unused batteries, computing of time-left and low battery alerts when there&rsquo;s more than one battery available, and handling of battery disconnects while running.</p>
<p>jscipione made some code changes to Tracker related to a work-in-progress refactor of the shortcuts code. waddlesplash made a number of cleanups to the Find panel, deleting unneeded and unwanted code especially around the &ldquo;clean up old queries&rdquo; feature, and reinstating the draggable query icon. jscipione fixed a variety of issues around various parts of Tracker, including automatically closing extra windows when switching to &ldquo;spatial&rdquo; mode, fixing crashes related to popup menus, and more.</p>
<p>CodeforEvolution implemented clearing the scrollback via an ANSI control sequence in Terminal (a feature already supported by xterm, Konsole, and others.)</p>
<h3 id="kits">Kits</h3>
<p>nipos fixed the Calendar view to vertically center date labels.</p>
<p>waddlesplash added some checks in BMessage to prevent more crashes when deserializing corrupt messages.</p>
<p>mt fixed some argument naming inconsistencies spotted by cppcheck.</p>
<p>X512 cleaned up some code in BView to use a utility function for sending <code>BShape</code> data to app_server.</p>
<p>ayu-ch changed <code>BPoint::PrintToStream</code> to actually print some decimal places.</p>
<p>nephele changed <code>BTextControl</code> to make the background of controls in an &ldquo;invalid&rdquo; state turn red.</p>
<h3 id="drivers">Drivers</h3>
<p>waddlesplash adjusted the <code>broadcom570x</code> driver to use MSIs for interrupts when possible, fixing support for various devices.</p>
<p>CodeforEvolution ported the <code>vmxnet</code> driver from FreeBSD, to support VMware&rsquo;s <code>VMXNET</code> paravirtualized networking interface.</p>
<p>waddlesplash adjusted how reference counting for USB objects works, fixing some bugs and adding more robust assertions (though this introduced some new KDLs which could either be old bugs or just new regressions which haven&rsquo;t been solved just yet.)</p>
<p>gscrain added some missing definitions from the USB Audio v2 specification to the headers, and added more dumping of its descriptors to <code>listusb</code>. (The USB audio driver itself hasn&rsquo;t been modified yet to support this.)</p>
<p>waddlesplash renamed the <code>intel_cstates</code> CPU idle driver to <code>x86_cstates</code>, as it also works on modern AMD hardware, and adjusted some fallback behavior in it. (He also modified another spin-loop in the kernel to make use of it.)</p>
<p>waddlesplash added some more sanity assertions to the network stack.</p>
<p>waddlesplash removed some duplicated code from the AHCI (SATA) driver, making it use common routines instead. He also fixed a 32/64-bit address mixup, which was causing boot failures on some 32-bit systems with more than 4GB of RAM.</p>
<p>waddlesplash reduced some log spam from the HDA driver.</p>
<p>waddlesplash fixed the SCSI stack to more properly respect DMA address restrictions.</p>
<h3 id="file-systems">File systems</h3>
<p>waddlesplash optimized searching for the next free block in the BFS block bitmap code, greatly speeding up creation and resizing of files on especially fragmented partitions.</p>
<p>waddlesplash reworked packagefs to have only installed directory nodes be r/w-locked, instead of <em>all</em> nodes. This saves 40 bytes (on 64-bit systems) per installed package file, which adds up to multiple MB on a system with lots of packages installed.</p>
<p>waddlesplash cleaned up some unused parameters in the block cache APIs.</p>
<h3 id="libroot--kernel">libroot &amp; kernel</h3>
<p>waddlesplash cleaned up the bootloader&rsquo;s heap memory allocation logic, and then increased the size of maximum &ldquo;standard&rdquo; heap allocations to be large enough for packagefs buffers to fit. This massively reduces after-boot memory usage when booting with the BIOS loader, as not all bootloader heap memory is properly cleaned up yet, it seems. (The EFI loader is also affected, but not nearly as badly.) He also tried to refactor some of the bootloader memory management code to make memory cleanups on exit more feasible, but encountered a number of difficulties and only partially succeeded.</p>
<p>VoloDroid fixed the re-initialization of serial output in the EFI bootloader on x86_64.</p>
<p>waddlesplash turned down tracing in the SMP code in the BIOS bootloader.</p>
<p>waddlesplash adjusted the bootloader to display the system package version with the latest state, making it clearer what will actually be booted.</p>
<p>waddlesplash relaxed a check in the virtual memory types logic when legacy MTRRs are enabled, avoiding KDLs in some situations.</p>
<p>waddlesplash made the kernel discard swap space on-disk when swap is disabled.</p>
<p>waddlesplash added an assertion in the core virtual memory code to check that we don&rsquo;t commit (reserve) more memory than we need for any given region, and fixed a number of places in the code where the accounting for this wasn&rsquo;t quite correct (though some still linger.) He also fixed interaction between the &ldquo;set memory protection&rdquo; (<code>mprotect</code>) and &ldquo;set area protection&rdquo; APIs. He also cleaned up some of the assertions and KDL commands, implemented some more debugging methods, and moved some source code around for better organization.</p>
<p>waddlesplash did some refactors and cleanups to the early boot memory allocation code in the kernel, fixing the boot on some systems with large amounts of RAM (or systems with unusual memory layouts.)</p>
<p>waddlesplash (re)added some more assertions to the kernel r/w locking code, fixed some logic errors in the low resource manager, and added a missing vnode reference acquisition in the file cache.</p>
<p>mmlr implemented Ctrl+D as a shortcut to exit KDL (when it&rsquo;s possible to do so, that is.)</p>
<p>nipos and waddlesplash fixed some bugs in the ICU time conversion code for dates in the future (or past).</p>
<p>waddlesplash adjusted more parts of libroot to use the common utility function for converting <code>timespec</code> structures.</p>
<p>waddlesplash cleaned up macros in some headers, including <code>&lt;sys/select.h&gt;</code>, to avoid namespace pollution and duplicate definitions.</p>
<p>korli adjusted FIFOs to honor <code>O_NONBLOCK</code> on <code>open()</code> correctly.</p>
<p>korli changed driver loading to prefer loading from the non-packaged / &ldquo;user&rdquo; directories over the packaged ones (unless loading drivers from non-packaged is disabled entirely.)</p>
<h3 id="documentation">Documentation</h3>
<p>drea233 fixed a typo in the Keyboard documentation.</p>
<h3 id="build-system">Build system</h3>
<p>jessicah made <code>./configure</code> detect the system <code>wget</code>&rsquo;s supported retry flags. waddlesplash adjusted how a Python interpreter can be specified to <code>./configure</code> to match other tools, and fixed its version checks. OscarL made <code>./configure</code> automatically choose the right GCC when running on <code>x86_gcc2h</code> platforms.</p>
<p>CodeforEvolution removed an obsolete definition from the <code>package_repo</code> Jamfile.</p>
<p>waddlesplash rehabilitated the build of the <code>DiskDeviceManagerTest</code>.</p>
<h3 id="arm">ARM</h3>
<p>oanderso implemented instruction cache synchronization, fixed some double-locking problems, implemented <code>VMTranslationMap::MaxPagesNeededToMap</code>,</p>
<h3 id="thats-all-folks">That&rsquo;s all, folks!</h3>
<p>Thanks again to all who contribute to Haiku, and especially those donors who make my contract possible.</p></description></item><item><title>Haiku Activity & Contract Report, September 2024 (ft. packagefs memory usage optimization)</title><link>https://www.haiku-os.org/blog/waddlesplash/2024-10-11-haiku_activity_contract_report_september_2024/</link><author>waddlesplash</author><pubDate>Fri, 11 Oct 2024 15:30:00 -0400</pubDate><guid>https://www.haiku-os.org/blog/waddlesplash/2024-10-11-haiku_activity_contract_report_september_2024/</guid><description><p>This report covers hrev58043 through hrev58187.</p>
<p>We&rsquo;re now back to development &ldquo;as usual&rdquo; after the release of R1/beta5 (though some of the changes in this report did make it in to the release itself.)</p>
<h3 id="packagefs-memory-usage-optimization">packagefs memory usage optimization</h3>
<p>While waiting for beta5-related rebuilds and other work to be completed last month, I spent a bunch of time trying to optimize packagefs&rsquo;s memory usage. There were a variety of changes here, such as: use singly- instead of doubly-linked lists where no performance penalty would be incurred, drop unused or constant fields from classes, inline reference counts for more optimal structure packing, eliminate unneeded padding, drop &ldquo;object depots&rdquo; from allocation arenas, make more use of allocation areas where it makes sense to, reduce locking granularity, implement and use a &ldquo;bump allocator&rdquo; for very short-lived objects, and more.</p>
<p>(Also while working on packagefs, I noticed our default string hashing routine was a somewhat outdated one; I replaced it with the now-common <code>hashdjb2</code> not just for packagefs but for most of the kernel, and then all of userspace as well.)</p>
<p>The final result is savings of somewhere around (or possibly more than) 20% of total packagefs memory usage, excluding file data caches, which amounts to at least 15 MB on my system.</p>
<p>Now, on to the rest of the report.</p>
<h3 id="applications">Applications</h3>
<p>jscipione cleaned up some code in Tracker, and fixed both Deskbar and Tracker to use the &ldquo;Menu&rdquo; font in more menus. OscarL fixed a Tracker crash on screen resolution changes. humdinger made the text input column in Tracker&rsquo;s Find panel wider, and jscipione removed an extra prompt to the user when dragging items out of query windows.</p>
<p>apl fixed the text colors in the &ldquo;Usage conditions&rdquo; window in HaikuDepot. jscipione fixed a crash when clicking on the conditions window before it&rsquo;s finished loading.</p>
<p>waddlesplash updated the list of maintainers in AboutSystem&rsquo;s credits.</p>
<p>nipos adjusted WebPositive&rsquo;s font settings preferences to support word wrapping. He also adjusted Keymap&rsquo;s buttons to use the standard system control colors, fixed Mail to make quoted text colors readable in dark mode, and fixed the text color in Tracker&rsquo;s &ldquo;File permissions&rdquo; and &ldquo;Status&rdquo; views.</p>
<p>korli adjusted the Intel partition map editor to automatically enable the &ldquo;Active&rdquo; checkbox for the first primary partition.</p>
<p>korli adjusted the default &ldquo;Mute&rdquo; media key shortcut to toggle mute on and off instead of just muting.</p>
<p>dovsienko fixed the Clock desktop applet pointing its hands imprecisely.</p>
<p>korli fixed ActivityMonitor to not start offscreen after restoring its position.</p>
<p>humdinger fixed quoting of partition names in some DriveSetup messages.</p>
<h3 id="command-line-tools">Command line tools</h3>
<p>An old patch from kallisti5 to improve <code>launch_roster</code>&rsquo;s output (displaying more information by default, in a table format, including service status) was revamped and merged by waddlesplash.</p>
<p>waddlesplash added support for printing the mutex option flags (e.g. <code>MUTEX_SHARED</code>, for mutexes shared across processes) to <code>strace</code>.</p>
<p>waddlesplash added a <code>-f</code> option to <code>package_repo list</code> to print the canonical filenames of packages in a given <code>repo</code> file. (This is useful for certain tools that want to download packages outside of Haiku.)</p>
<h3 id="servers">Servers</h3>
<p>jscipione fixed the colors of the &ldquo;problem&rdquo; and &ldquo;results&rdquo; windows in package_daemon (the ones that appear when there are problems installing or updating packages in HaikuDepot or SoftwareUpdater.)</p>
<p>waddlesplash added some more checks to the audio mixer startup, to avoid hangs in more cases.</p>
<p>madmax adjusted input_server to avoid locking the device list in more cases, fixing hangs and deadlocks when device threads hang.</p>
<p>madmax fixed single-clicks on the resize corner of windows erroneously minimizing the window.</p>
<h3 id="kits">Kits</h3>
<p>jscipione fixed live updating of text colors in Spinner controls, adjusted how parent colors are adopted across most controls, and tweaked the button background color in &ldquo;dark mode.&rdquo;</p>
<p>waddlesplash made some adjustments to the Locale Kit to avoid crashing if ICU failed to initialize in more cases.</p>
<p>waddlesplash added another assertion to <code>BTimeSource</code> that published times don&rsquo;t have negative drift.</p>
<h3 id="drivers">Drivers</h3>
<p>tmtfx enabled support for &ldquo;Gemini Lake&rdquo; and &ldquo;Ice Lake&rdquo; devices in the <code>intel_extreme</code> modesetting driver. PulkoMandy added some more code to support &ldquo;Tiger Lake&rdquo;, and ilzu added some more &ldquo;Tiger Lake&rdquo; devices.</p>
<p>waddlesplash reduced the syslog spam of the <code>usb_audio</code> driver.</p>
<p>oanderso adjusted the ACPI driver to map its physical memory as &ldquo;write-back&rdquo; instead of &ldquo;uncached&rdquo;. This was necessary for the ARM64 port to work (as otherwise we get faults on unaligned addresses), but it is supported on x86 as well. This exposed some other bugs in Haiku related to double-mapping of such physical memory, which must use the same memory type in order to not encounter problems (as if we don&rsquo;t, we&rsquo;ll get &ldquo;machine check exception&quot;s.)</p>
<p>mmlr fixed some bugs related to allocation of queues in the virtio drivers, and added more error checks and handling. korli added a check for the device status when resetting virtio devices, and fixed an out-of-bounds read in the virtio_pci driver.</p>
<h3 id="file-systems">File systems</h3>
<p>waddlesplash did some minor code and terminology cleanup in BFS.</p>
<p>Jim906 fixed a crash in FAT when disk I/O fails in some cases. He also enabled R/W access to volumes up to 2 TB in size, and added checks to prevent overflowing the sector count.</p>
<h3 id="libroot--kernel">libroot &amp; kernel</h3>
<p>Some minor fixes and performance improvements to the kernel derived from changes by jpelczar were merged by waddlesplash, including to the kernel <code>malloc</code>, locking granularity in the VM, page reservation leaks and dangling reference detection, and more. He cleaned up some of the code around the merged changes, while at it.</p>
<p>waddlesplash fixed an offset computation bug in the kernel IORequest subsystem that was leading to failed I/O operations spuriously being reported as successful.</p>
<p>waddlesplash cleaned up the <code>*at</code> (e.g. <code>linkat</code>) function implementations in <code>libroot</code> and related modules, making them use the <code>AT_FDCWD</code> constant rather than hard-coding values and reducing code duplication. He then adjusted the kernel to only accept <code>AT_FDCWD</code> and <code>-1</code> (the old value of <code>AT_FDCWD</code>) for such syscalls, rather than any negative value.</p>
<p>waddlesplash replaced some custom-written libroot I/O code with glibc&rsquo;s. (Eventually we want to switch entirely to musl, but the stdio implementation is still from glibc, so we should use glibc&rsquo;s code here to be consistent for the moment.)</p>
<p>waddlesplash fixed <code>gmtime()</code> to return the proper &ldquo;GMT&rdquo; timezone string.</p>
<p>waddlesplash fixed conversion of surrogate pairs in <code>mbrtowc</code>.</p>
<p>waddlesplash made a number of fixes to the pthreads implementation, including adjusting error codes to be more in line with the POSIX specification, suppressing <code>EINTR</code> in rwlocks, fixing deadlocks in barriers, and more. He also fixed invalid timeouts being passed to the POSIX unnamed semaphore routines, and fixed a logic error leading to shared unnamed semaphores not working properly.</p>
<p>korli adjusted the kernel&rsquo;s linker scripts to warn or error if there are unexpected or unhandled sections.</p>
<p>waddlesplash fixed accounting for a currently running thread&rsquo;s CPU time, which should improve the display of thread CPU usage in various applications.</p>
<p>waddlesplash refactored syscall runtime accounting, to count only the time actually spent in the syscall, and not its total overhead (which we can&rsquo;t easily measure from within the kernel, anyway.) This makes the times in <code>strace</code> output much more useful to work with.</p>
<p>waddlesplash reduced the maximum size of dirent buffers allocated by the kernel to 8 KB. (Most things that read dirents never read anything as large as the old or new limits, and the things that did should actually be faster this way anyway, as we now always go through a faster allocation facility in the kernel instead of a slower one.) He adjusted the few places in the tree that used such large buffers to use the new limit instead.</p>
<p>waddlesplash adjusted the &ldquo;low resource&rdquo; subsystem to not timeout due to a race condition when waiting for a single run to complete.</p>
<p>waddlesplash added some more assertions to the kernel mutex facilities to catch more spurious wakeups. He also fixed lock ABI compatibility between KDEBUG and non-KDEBUG builds, which had been a problem for many years. Now, drivers and kernel add-ons built against KDEBUG kernels should work with non-KDEBUG ones, while drivers built against non-KDEBUG kernels will encounter &ldquo;missing symbol&rdquo; errors when you try to use them against KDEBUG kernels. (Previously one would just get hangs, deadlocks, or crashes when attempting to do either of those things.)</p>
<p>waddlesplash adjusted the <code>kqueue</code> implementation to use a common facility for converting timespecs, and added a check for invalid timespecs.</p>
<p>waddlesplash fixed a race condition leading to spurious wakeups in the thread unblock-after-timeout logic (a recent regression).</p>
<p>waddlesplash made some improvements to the VFS entry cache, most notably including caching of hash values, allowing the cache to skip string comparisons when the hash values don&rsquo;t match. On one simple benchmark (rebuilding HaikuDepot) it saved about 10,000 string comparisons.</p>
<p>waddlesplash added some more debugging information to the <code>slab_object</code> KDL command.</p>
<p>waddlesplash made some more adjustments to the rwlocks debugging code, including adding another &ldquo;wasn&rsquo;t locked&rdquo; assertion (that code isn&rsquo;t enabled by default even on <code>KDEBUG</code> builds, as it&rsquo;s part of the checks that are rather expensive to run) as well as a &ldquo;isn&rsquo;t read-locked&rdquo; assertion (on destruction, which is enabled on all builds.) He fixed a few minor issues these assertions uncovered.</p>
<p>waddlesplash re-enabled the &ldquo;initialize TSC from CPUID&rdquo; code for hypervisors. (It was already enabled for bare metal.)</p>
<p>waddlesplash adjusted <code>runtime_loader</code> to be built without <code>-fno-builtin</code>, the same as most of libroot, providing a small performance improvement.</p>
<p>mmlr implemented PAT (Page Attribute Table) support for x86, which supersedes MTRRs and allows us to disable them entirely on supported hardware. He also cleaned up some of the code surrounding MTRRs and PAT, added some new debug commands, and added assertions to detect overlapping memory regions with differing types, which fixed most of the issues uncovered/caused by the ACPI memory type change. waddlesplash then renamed the <code>B_MTR_*</code> constants to &ldquo;something more meaningful&rdquo; (as an old TODO comment indicated they should be).</p>
<p>korli implemented printing of more x86 CPU features in the debug output.</p>
<p>waddlesplash changed most remaining non-POSIX include/feature guards in standard headers to be <code>_DEFAULT_SOURCE</code>, so that our <code>&lt;features.h&gt;</code> can turn them on automatically in most cases.</p>
<p>waddlesplash added a missed check for the new-style <code>BlockedEntries</code> settings in the bootloader.</p>
<p>waddlesplash cleaned up some code and adjusted how locking works in some parts of the PS/2 driver, to try and fix some race conditions.</p>
<p>waddlesplash added some more information to the &ldquo;supposed to be free page&rdquo; panic, and adjusted the &ldquo;page&rdquo; KDL command to use proper debug facilities for enumerating over address spaces rather than brute-force iteration (which is too slow to work well on 64-bit address spaces.)</p>
<p>kallisti5 fixed the build of the kernel guarded heap.</p>
<h3 id="documentation">Documentation</h3>
<p>kallisti5 updated the instructions for generating Google Cloud Platform instances for Haiku.</p>
<h3 id="build-system">Build system</h3>
<p>oanderso fixed building the DNS resolver under Clang.</p>
<p>waddlesplash adjusted the build system to only build the <code>haiku_source</code> package if <code>HAIKU_INCLUDE_SOURCES</code> is set, decreasing the time of full <code>@nightly</code>-profile image builds.</p>
<p>waddlesplash synchronized the set of packages from HaikuPorts that Haiku is built against (primarily for the sake of the R1/beta5 release.)</p>
<p>jessicah added some missing checks and options for the Python tool in <code>./configure</code>, plus some sanity checks for cross-tools options.</p>
<p>waddlesplash moved the libroot malloc implementations to their own subdirectory (to pave the way for the addition of more allocator(s) in the future, most notably OpenBSD&rsquo;s malloc, which is a current candidate for replacing hoard2.)</p>
<p>PulkoMandy added Wonderbrush to the release images on all architectures, not just x86 32-bit.</p>
<p>waddlesplash unified the BeOS ABI compatibility preprocessor definitions to a single definition, and adjusted all code to match. He also disabled &ldquo;BeOS compatible types&rdquo; (for <code>int32</code>, etc.) in kernel mode, and fixed the code that had depended on this.</p>
<p>waddlesplash cleaned up the base C/C++ compiler flags in ArchitectureRules and streamlined the comments around them.</p>
<h3 id="arm">ARM</h3>
<p>oanderson continued his work on ARM64 support, refactoring and implementing many missing parts of the VM translation map, exception handling, and TLBs, among other things.</p>
<p>waddlesplash added some &ldquo;unimplemented&rdquo; prints to the default base class implementation of certain VM debug methods, to aid in finding where missing functionality needs to be implemented.</p>
<h3 id="thats-all-folks">That&rsquo;s all, folks!</h3>
<p>Thanks again to all who contribute to Haiku, and especially those donors who make my contract possible.</p></description></item><item><title>Haiku Activity & Contract Report, August 2024</title><link>https://www.haiku-os.org/blog/waddlesplash/2024-09-17-haiku_activity_contract_report_august_2024/</link><author>waddlesplash</author><pubDate>Tue, 17 Sep 2024 23:30:00 -0400</pubDate><guid>https://www.haiku-os.org/blog/waddlesplash/2024-09-17-haiku_activity_contract_report_august_2024/</guid><description><p>This report covers hrev57901 through hrev58042.</p>
<p>R1/beta5 was of course <a href="https://www.haiku-os.org/news/2024-09-13_haiku_r1_beta5/">released</a> just a few days ago, and many (though not all) of the changes in this report made it in to the release.</p>
<h3 id="applications">Applications</h3>
<p>madmax fixed a crash in AboutSystem caused by an incorrectly added copyright entry.</p>
<p>apl made a number of refactors to HaikuDepot&rsquo;s code, including to the language model, path handling, package data fetching, the display of publishers, and more.</p>
<p>nipos and nephele submitted fixes to the ColumnListView, Debugger, WebPositive, ResEdit, FontDemo, Cortex, Sudoku, and more to remove hard-coded colors and make things look better in &ldquo;dark mode&rdquo;. (They also adjusted the computation used in Appearance for some colors in &ldquo;dark mode&rdquo; to make them fit in better.)</p>
<p>waddlesplash fixed the disassembler used by Debugger to handle addresses coming from registers in address computations (and submitted the fix upstream, as this is a third-party library.)</p>
<p>jscipione adjusted Tracker to repopulate menus when entering or leaving volumes, and to properly enable and disable some items in the &ldquo;New&rdquo; menu whenever menus are updated. He also fixed the &ldquo;Copy&rdquo; and &ldquo;Cut&rdquo; actions to not alter the selection (and fixed some other issues around Move and Copy, related to redraws), fixed some issues with colors in the &ldquo;Open with&hellip;&rdquo; window, and made drag-and-drops from virtual folders create symbolic links (instead of copying the underlying file.)</p>
<p>humdinger fixed some bugs in Tracker relating to the naming of duplicating or creating new files, including around locale awareness. He also added an option to create a &ldquo;New template folder&rdquo;.</p>
<p>humdinger tweaked some GUI strings in various applications.</p>
<p>CalistoMathias (a GSoC &lsquo;24 student)&rsquo;s changes to implement basic folder filtering in Tracker queries were merged (however, this was done after the beta5 release was branched, so this feature isn&rsquo;t in the release.) jscipione later came by did a few cleanups to it.</p>
<p>nephele fixed the build of &ldquo;Switcher&rdquo;.</p>
<p>OscarL fixed Clock to ensure that the first draw uses the current time (not some default time.)</p>
<p>jscipione fixed status icons not appearing in the &ldquo;Modifier Keys&rdquo; window in Keymap.</p>
<h3 id="command-line-tools">Command line tools</h3>
<p>waddlesplash added logic to <code>profile</code> to report the count of events dropped by the system profiler.</p>
<p>mmlr added an argument to <code>package_repo</code>&rsquo;s &ldquo;update&rdquo; command that allows it to be used in an environment where not all packages in the repository are locally present (e.g. they might be stored in S3 buckets instead.)</p>
<h3 id="servers">Servers</h3>
<p>madmax added some missing locks of the font manager to app_server, and fortified the logic around addition and removal of fonts. He also fixed the font manager to not scan all directories twice on startup.</p>
<p>waddlesplash fixed a (probably harmless) broken timeout computation in app_server (uncovered in the process of working on the kernel timers change mentioned below.)</p>
<p>jscipione fixed window tab invalidation logic in app_server, fixing some very old &ldquo;tab still appears after being removed from the stack&rdquo; redraw glitches.</p>
<p>waddlesplash fixed a number of time computation bugs in the media mixer (uncovered by the <code>BTimeSource</code> assertions change mentioned below), which fixed many (if not most or all) of the cases of the &ldquo;no audio immediately after boot, only some time later&rdquo; bug. He also improved its event scheduling efficiency, fixed some of its logic around disconnecting/reconnecting to outputs, and fixed some crashes in the multi-audio node caused by certain audio devices reporting invalid control types. (He also experimented with adding logic to switch sound outputs without restarting the media services, but this doesn&rsquo;t work quite yet.)</p>
<p>waddlesplash added a check in registrar that clipboard download messages are sent properly, and added error handling for when they don&rsquo;t. (This, along with a corresponding change in <code>libbe</code>, fixes some application hangs on low-memory conditions when there&rsquo;s an especially large clipboard.)</p>
<p>madmax added some checks in input_server to avoid invalid mouse settings files causing the mouse to not work at all.</p>
<p>waddlesplash moved some of the font-management code around in <code>app_server</code> to avoid unnecessary waits for the font manager&rsquo;s thread during startup, when possible.</p>
<p>PulkoMandy and X512 fixed a conflict between the &ldquo;power&rdquo; keycode and the Japanese <code>\_</code> key, and adjusted the keymap documentation.</p>
<p>waddlesplash changed Debugger to not query the Package Kit in non-interactive mode at all (and fixed some other bugs, in Debugger and the Package Kit, related to this that were causing crashes or even hangs in various circumstances.)</p>
<p>mmlr fixed a use of uninitialized memory in the DHCP client (discovered by using the guarded heap.)</p>
<h3 id="kits">Kits</h3>
<p>Zardshard made a change to have BBitmaps with the <code>ACCEPTS_VIEWS</code> flag set not be cleared automatically, fixing a compatibility issue with BeOS (as well as making the use of such bitmaps more convenient in multiprocess applications like WebKit.)</p>
<p>waddlesplash added an assertion in <code>BTimeSource</code> in the Media Kit to prevent applications and services from trying to publish times with &ldquo;drift&rdquo; factors of 0. The &ldquo;drift&rdquo; value is used as a multipler and a divisor to correct for time/clock drift, so a value of zero (or negative) produces broken results. This was uncovered by the kernel timers change mentioned below, and it in turn uncovered more bugs in the media add-ons code.</p>
<p>waddlesplash fixed an API compatibility regression in <code>BListView</code> that was causing stack overflow crashes on drag-and-scroll in certain applications (like WonderBrush). jscipione adjusted <code>BListView</code> to not update selections on mouse up when dragging.</p>
<p>PulkoMandy fixed <code>BMenu</code> to draw checkmarks properly on matrix-layout menus.</p>
<h3 id="drivers">Drivers</h3>
<p>waddlesplash implemented receive checksum offloading for IP, TCP, and UDP in <code>virtio_net</code>, and fixed some logic and buffer sizing errors in the driver as well.</p>
<p>korli added definitions and logic for &ldquo;SuperSpeedPlus&rdquo;, a revision of USB3 that allows for 10 Gbps connections.</p>
<p>waddlesplash revised some of the out-of-order ACK logic in the TCP module, adjusted some debugging code to help with diagnosing some intermittent KDLs that some users have reported, and tweaked the window-update sending logic to avoid sending window updates unnecessarily. This was all in preparation for the implementation of dynamic receive window sizing, which was also merged last month (along with related fixes to the round-trip-time calculation), resulting in massively improved TCP throughput (as much as 10x or more depending on the connection.) Users who have previously seen slow download speeds on Haiku over long-distance connections should see very significant improvements now.</p>
<p>waddlesplash cleaned up some code and constants around ethernet MTUs, fixing some incorrect sizing in a few places. He then disabled MTUs larger than the ethernet maximum, as these aren&rsquo;t handled properly at the moment and just caused traffic loss. (In order to handle these properly, we need to implement &ldquo;Path MTU Discovery&rdquo;, which at present we don&rsquo;t.)</p>
<p>PulkoMandy fixed boundary checks and adjusted read/write logic in the &ldquo;Enhanced&rdquo; (generally ACPI) PCI bus&rsquo;s configuration space code, to resolve &ldquo;boot failure&rdquo; regressions following the PCI refactors.</p>
<p>waddlesplash turned down some tracing constants to avoid syslog spam in some drivers.</p>
<p>waddlesplash fixed some bugs in the allocation of DMA buffers for FreeBSD/OpenBSD network drivers. He also fixed an incorrect buffer size in the receive path, which was a recent regression that was causing certain drivers not to work at all on some hardware.</p>
<p>waddlesplash made some slight efficiency improvements to the NVMe driver.</p>
<p>PulkoMandy updated ACPICA to a version from this March, and fixed a few minor issues in our code that interfaces with it.</p>
<p>waddlesplash cleaned up a variety of code related to ancillary data in the UNIX domain sockets module, and fixed some minor memory leaks in the network stack related to it. He also fixed a relatively rare KDL that sometimes happened when closing such sockets.</p>
<h3 id="file-systems">File systems</h3>
<p>waddlesplash unified some previously duplicated code between <code>ramfs</code> and <code>ram_disk</code>, made a few minor adjustments for efficiency, and then fixed it to allocate &ldquo;cleared&rdquo; pages. This fixes a violation of the POSIX specification that Clang depended on when working with memory-mapped files off <code>ramfs</code>. He also cleaned up and unified some code, adjusted the memory reservation priority (to avoid whole-system hangs on low-memory conditions), and fixed some missing sets in <code>ramfs</code> that were leading to KDLs.</p>
<p>waddlesplash added some more locking assertions to packagefs (and fixed the minor bugs they uncovered), and then added better diagnostic messages for the case where conflicting packages are requested to be mounted.</p>
<p>Jim906 fixed the initialization of media bytes in the FAT driver, made a change to speed up mounting of large volumes, and fixed errors in the handling of volume labels.</p>
<p>waddlesplash turned a kernel panic in the common query parser into a properly reported error.</p>
<p>waddlesplash started cleaning up the use of memory arenas (<code>object_cache</code>s) in packagefs, in preparation for changes reducing its overall memory usage.</p>
<h3 id="libroot--kernel">libroot &amp; kernel</h3>
<p>waddlesplash cleaned up some of the code in <code>WeakReferenceable</code>, an intrusive reference-counting class that allows for &ldquo;weak&rdquo; references.</p>
<p>waddlesplash cleaned up some logic in the kernel thread code to not rely on certain return values to cancel timeouts (though this change introduced a regression that wasn&rsquo;t fixed till this month.) He also fixed the scheduling of &ldquo;absolute-real-time timeouts&rdquo;; timeouts that have a time relative to the real time clock (instead of the system uptime clock.) These had previously only worked by chance (and probably in some cases would&rsquo;ve gotten stuck for longer periods of time than they should have randomly). He performed some cleanups of the timer code at the same time.</p>
<p>waddlesplash fixed a check in the kernel&rsquo;s user-debugger code (the kernel interface used by the Debugger application) that was leading to whole-system hangs when the <code>debug_server</code> crashed or hung in certain circumstances (such as low memory.)</p>
<p>waddlesplash added some more sanity checks in the VFS to catch misbehaving filesystem drivers earlier, rather than causing cryptic KDLs later.</p>
<p>waddlesplash added a new mode to the &ldquo;boot profiler&rdquo; to allow the <code>scheduling_recorder</code> to capture the boot process, allowing thread latencies and lock contention during it to be analyzed. (He also fixed some other miscellaneous issues surrounding this feature in the various tools related to it: scrollbars in DebugAnalyzer, kernel permissions in <code>transfer_area</code>, etc.)</p>
<p>waddlesplash removed an unneeded scan of the SCSI bus and drivers from the device manager, reducing boot times somewhat.</p>
<p>waddlesplash fixed a major oversight in the kernel&rsquo;s <code>IORequest</code> API that was leading to mixups between the total size of a request and the last offset it transferred (i.e. the starting offset plus the total size.) This was leading to buffer overruns in the file cache, causing KDLs and other problems with cryptic errors about page structures. (He later added more assertions to the page code to catch misuses of the page structures like this more easily in the future.)</p>
<p>waddlesplash added an assertion to the kernel&rsquo;s interrupt handling logic to catch a bug in VM integration driver, and some checks in the global areas tree to catch duplicate area IDs.</p>
<p>waddlesplash fixed a bug in the <code>kqueue</code> implementation that was leading to <code>libevent</code>&rsquo;s <code>kqueue</code> backend hanging on startup.</p>
<p>waddlesplash made a change to the kernel on 32-bit x86 to report syscall return values to <code>strace</code> more correctly (however, it wound up introducing a bug that caused intermittent crashes of many applications which wasn&rsquo;t discovered and fixed till near the end of the month.)</p>
<p>waddlesplash fixed some crashes in <code>runtime_loader</code> that could happen when not all the dependencies of a dynamically-loaded library were present. He also fixed some reference counting and cache management bugs between <code>ramfs</code> and the kernel VFS. (These two issues were uncovered by the unstable, experimental port of Firefox.)</p>
<p>waddlesplash renamed some architecture-specific functions introduced on x86 to have more generic names, as they&rsquo;re now declared and used on other architectures as well.</p>
<p>waddlesplash cleaned up some BeOS compatibility code in <code>libnetwork</code>, and adjusted things so that it&rsquo;s not even compiled in on most architectures.</p>
<p>waddlesplash implemented the one portable case of <code>mknod</code>/<code>mknodat</code> (the one where it&rsquo;s equivalent to <code>mkfifo</code>), as defined in POSIX.</p>
<p>mmlr made some fixes and cleanups to the guarded heap in <code>libroot_debug</code>, allowing it to be used in more scenarios (potentially on the whole system at once.)</p>
<h3 id="documentation">Documentation</h3>
<p>waddlesplash adjusted the definition of <code>B_INFINITE_TIMEOUT</code> to make clearer that it&rsquo;s <code>INT64_MAX</code> (and cleaned up some code related to it.)</p>
<p>humdinger added a section to the HIG on &ldquo;Recently Used Files&rdquo;.</p>
<p>waddlesplash renamed the &ldquo;Release Milestones&rdquo; developer documentation page to &ldquo;Release Cookbook&rdquo;, which was what it used to be called when it was on Trac (and more accurately reflects what&rsquo;s in it.) He then updated some of its sections for clarity.</p>
<h3 id="build-system">Build system</h3>
<p>X512 submitted a patch to put each build target on a separate line, in large part to make rebases and resolving merge conflicts in these files easier.</p>
<p>PulkoMandy, waddlesplash, and kallisti5 upgraded Haiku and its dependencies to be built against OpenSSL 3, Python 3.10, and a variety of other newer versions of standard packages.</p>
<p>PulkoMandy added a missing include of <code>&lt;features.h&gt;</code> in <code>&lt;sched.h&gt;</code>, to allow applications to use its extra features without needing to define any extra preprocessor macros.</p>
<p>waddlesplash updated the &ldquo;WebPositive bookmarks&rdquo; files, fixing some bugs in their redirections.</p>
<p>korli added the <code>linprog</code> library headers to the <code>haiku_devel</code> package. jmairboeck fixed some typos in comments in various compatibility headers, and waddlesplash fixed a C89 compatibility issue in one.</p>
<p>jscipione updated the size of nightly images to 650MB, so that there&rsquo;s a bit more free space available.</p>
<p>waddlesplash made a variety of changes around the tree and the build system to get Clang builds much closer to a working state again (and also cleaned up some code while at it.)</p>
<h3 id="arm--powerpc">ARM &amp; PowerPC</h3>
<p>zeldakatze fixed the text console in the OpenFirmware bootloader.</p>
<p>archeYR fixed the ARM kernel arch code to use the correct instruction for CPU idling.</p>
<p>oanderso fixed some bugs with cache and MMU maintenance in the ARM64 bootloader, allowing the boot to get into the kernel under virtualization on the Apple M1, then fixed the kernel timer code to work in a virtualized environment, and started making changes and implementing missing features in the MMU code.</p>
<h3 id="thats-all-folks">That&rsquo;s all, folks!</h3>
<p>August was quite a busy month! Thanks again to all who contribute to Haiku, and especially those donors who make my contract possible.</p></description></item><item><title>Haiku R1/beta5 has been released!</title><link>https://www.haiku-os.org/news/2024-09-13_haiku_r1_beta5/</link><pubDate>Fri, 13 Sep 2024 12:00:00 +0000</pubDate><guid>https://www.haiku-os.org/news/2024-09-13_haiku_r1_beta5/</guid><description><p>After about a year and a half since the last beta, Haiku R1/beta5 has been released. See &ldquo;<strong><a href="./get-haiku/r1beta5/release-notes/">Release Notes</a></strong>&rdquo; for the release notes, &ldquo;<a href="./get-haiku/r1beta5/release-notes/#press-contact">Press contact</a>&rdquo;, for press inquiries &hellip; and &ldquo;<strong><a href="./get-haiku/r1beta5/">Get Haiku!</a></strong>&rdquo; to skip all that and just download the release (or upgrade to it from an existing install!)</p></description></item><item><title>[GSoC 2024]: Implementing Incremental Search [Final Report]</title><link>https://www.haiku-os.org/blog/calisto-mathias/2024-08-24_gsoc_2024_implementing_incremental_search_final_report/</link><author>calisto-mathias</author><pubDate>Sat, 24 Aug 2024 01:46:45 +0530</pubDate><guid>https://www.haiku-os.org/blog/calisto-mathias/2024-08-24_gsoc_2024_implementing_incremental_search_final_report/</guid><description><h2 id="introduction">Introduction</h2>
<p>The goal of this post is to document the changes I’ve successfully made during the GSoC period, the current state of the project, future enhancement goals, and a few other topics. I also want to extend my thanks to the Haiku developers and community for the opportunity to work on this fantastic operating system.</p>
<h2 id="background">Background</h2>
<p>Haiku is a truly innovative operating system. One of its most interesting approaches to a filesystem lies in the fact that it makes metadata a primary characteristic of the filesystem. In fact, this was one of the core features that it carried over from its predecessor BeOS.</p>
<p>So, what was my project about? Despite Haiku’s fast search capabilities, the Find Panel was outdated and lacked several features that could improve user experience. My research identified a few missing functionalities that could greatly enhance file searching:</p>
<ol>
<li>The ability to limit searches to specific directories.</li>
<li>The option to pause a running query.</li>
<li>A more user-friendly Find Panel interface, with search columns aligned to the attributes being queried.</li>
<li>Support for incremental search, allowing users to see search results update as they modify their search terms.</li>
</ol>
<p>These improvements are designed to make searching in Haiku even more powerful and intuitive.</p>
<h2 id="incremental-search-a-bit-more-of-an-explanation">Incremental Search: A Bit More of an Explanation</h2>
<p>Have you ever run a search and wanted to tweak your search terms repeatedly while seeing updates in real time? That’s what incremental search offers. It allows you to adjust search terms and see results update automatically, without having to start a new search or manually re-run the current one. Instead of executing and adjusting queries repeatedly, incremental search lets you type and modify terms in one go, updating results on the fly.</p>
<p>This was the central focus of my project and where I dedicated most of my efforts!</p>
<h2 id="what-i-accomplished-during-my-google-summer-of-code-period">What I Accomplished During My Google Summer of Code Period</h2>
<h3 id="cleaning-up-the-user-interface-of-the-find-panel">Cleaning Up the User Interface of the Find Panel</h3>
<p>My first task was to reorganize the cluttered options in the Find Panel into a dedicated menu bar. This introductory task helped me get acquainted with Haiku’s development style. Although progress was slow initially due to my inexperience, I learned a lot from this process. Here’s a look at the initial changes and their appearance:</p>
<p>Following is the Old Find Panel present in the Haiku Operating System:
<br><br>
<img src="./files/blog/calisto-mathias/Final-Report/old-find-panel.png" alt="Old Find panel">
<br><br>
And here is the new one:
<br><br>
<img src="./files/blog/calisto-mathias/Final-Report/new-find-panel.png" alt="New Find Panel" width="598px">
<br><br></p>
<h3 id="adding-folder-filtering-abilities-to-searches-executed-through-the-find-panel">Adding Folder Filtering Abilities to Searches Executed Through the Find Panel</h3>
<p>The second task involved adding folder-based filtering to searches via the Find Panel. This enhancement was implemented at the application layer rather than the file system level. It maintains high search speeds while limiting results to selected directories. Users can now select multiple directories and merge the search results within the results panel. This functionality is accessible through the third drop-down menu from the left.</p>
<p>Users can select directories using the last option in this drop-down menu. This action opens a file panel where they can choose one or more directories (or symbolic links to directories). The selected directories then appear in a clearly visible section within the same drop-down menu. Users also have the option to unselect any directories they no longer want to include.</p>
<p>We can now select these directories for search through the following drop-down menu as shown in the pictures below
<br><br>
<img src="./files/blog/calisto-mathias/Final-Report/find-panel-with-directory-selectors.png" alt="New Find Panel Directory Selector" width="598px">
<br><br>
<img src="./files/blog/calisto-mathias/Final-Report/Selecting-Directories.png" alt="New Find Panel Directory Selector" width="598px">
<br><br>
<img src="./files/blog/calisto-mathias/Final-Report/directories-entries-in-menu.png" alt="New Find Panel Directory Selector" width="598px"></p>
<h3 id="adding-the-design-for-an-incremental-search-find-panel">Adding the Design for an Incremental Search Find Panel</h3>
<p>The most crucial and challenging aspect of my project was implementing the design for the incremental search find panel, which associates attribute columns with their corresponding search columns. While I managed to implement the core functionality, there are a few limitations that need to be addressed in the future.</p>
<p>Currently, the incremental find panel integrated into the Find Panel results window is under review on <a href="https://review.haiku-os.org/c/haiku/+/8096">Gerrit</a>. It will take some time before it is merged into the main codebase. Despite the challenges, I managed to get the core functionality working, allowing users to pause and restart their queries, as well as adjust search terms, all within the results window without switching back and forth repeatedly.</p>
<img src="./files/blog/calisto-mathias/Final-Report/incremental-search-first-look.png" alt="Incremental Search panel">
<br><br>
The user can select the various methods of combination in the same column. This combination follows the same rules as the searcy "by-attribute" mode in the old find panel.
<img src="./files/blog/calisto-mathias/Final-Report/selecting-combination-options.png" alt="Incremental Search Panel">
<br><br>
<p>Apart from this, users can pause and re-execute their queries on the file-system without any hassle now. The columns can also be moved around with each other, in order to set more powerful combination modes. The only caveat at the moment is that the two different combination modes for the incremental find panel hasn&rsquo;t been completed. As of now, the columns are always combined using the &ldquo;AND&rdquo; operator (so the order does not matter, but this is temporary!)</p>
<h2 id="whats-still-left-to-do">What’s Still Left to Do?</h2>
<p>Implementing this project in one go was quite challenging. Initially, I hoped to complete everything by the project deadline, but that proved unrealistic. It took me some time to grasp the technical details involved. However, I believe I can complete the remaining work with a bit more time and by tackling it in smaller steps.</p>
<p>Here’s what remains to be done in the incremental find panel:</p>
<ol>
<li><strong>Incremental Search with Keystrokes:</strong> The goal was to refresh results with each keystroke. Although I implemented this, it turned out to be unstable due to the high load from frequent clearing, refreshing, and restarting of queries. I had planned to develop a debouncer to manage this load but didn’t manage to complete it. This will be my primary focus after the GSoC’24 period ends. A possible strategy to implement this that a future developer could look into with this could be to use a thread that continuously polls the predicate string. This would make use of synchronisation primitives and might be one way that we could tackle this issue. During my time in the GSoC period, I tried to make use of the BMessageRunner and BLooper classes to try this but I was unable to find a working strategy. I will keep working on this as well!</li>
<li><strong>Folder Filtering Integration:</strong> Folder filtering is not yet functional in the incremental search panel within the results window. This should be a straightforward addition, and I plan to incorporate it as soon as the initial code is merged.</li>
</ol>
<h2 id="learning-experience-from-gsoc24">Learning Experience from GSoC’24</h2>
<p>GSoC’24 with Haiku has been an incredibly rewarding experience. Before starting, I was relatively new to C++ and struggled with using git effectively. Thanks to GSoC, I’ve gained confidence and skills. Although I still have a long way to go, working on Haiku has inspired me to explore further and contribute more to both Haiku and other open-source projects.</p>
<p>Interacting with the Haiku community and receiving their advice has been invaluable. I feel like I’ve made some great friends along the way, which has made the experience even more enjoyable.
files/blog/calisto-mathias/Final-Report/
Here are some key lessons I’ve learned during my time with Haiku:</p>
<ol>
<li><strong>Git:</strong> Initially, I struggled with git, having only used it for small projects. Working with Gerrit taught me how to use git more effectively for collaboration. Special thanks to zardshard and waddlesplash for their guidance and resources. I particularly appreciated waddlesplash’s detailed explanation of git fundamentals.</li>
<li><strong>Importance of Style Guidelines:</strong> I never used a style guideline for my personal projects, but I’ve learned how crucial they are for development speed. The Haiku style guide has proven to be highly effective, and I plan to adopt it for all my future C++ projects.</li>
<li><strong>Navigating Large Codebases:</strong> At first, I found it challenging to work with a large codebase. I felt overwhelmed by its size and complexity. However, over time, I improved my ability to navigate and contribute to the codebase, including adding a new view to the Haiku Results panel.</li>
<li><strong>Communication Skills:</strong> This project has significantly enhanced my ability to communicate effectively with community members.</li>
</ol>
<h2 id="conclusion">Conclusion</h2>
<p>I’m deeply grateful to the Haiku community and development team for giving me the chance to contribute to Haiku. This project has not only improved my programming skills but also made me a fan of the Haiku operating system. I’m eager to keep contributing and learning as I continue my journey with Haiku.</p>
<p>A big thank you to my mentors, Niels Sascha Reedijk and Humdinger, for their ongoing support and guidance throughout the GSoC project. Their timely responses and insightful advice made the process smoother and helped me overcome many challenges and self-doubt.</p>
<p>I’m excited to keep contributing to the Haiku operating system and look forward to future opportunities to grow and learn.</p>
<h2 id="keeping-track-of-this-project">Keeping Track of this Project</h2>
<p>Haiku uses the Gerrit Revision Control system for keeping track of changes and commits. You can find the current status of my project on this changeset: <a href="https://review.haiku-os.org/c/haiku/+/8096">Haiku Review</a>!</p></description></item><item><title>[GSoC 2024] Virtio Sound: Final Report</title><link>https://www.haiku-os.org/blog/diegoroux/2024-08-22_gsoc_2024_final_report/</link><author>diegoroux</author><pubDate>Thu, 22 Aug 2024 18:39:13 -0600</pubDate><guid>https://www.haiku-os.org/blog/diegoroux/2024-08-22_gsoc_2024_final_report/</guid><description><h1 id="gerrit-submission">Gerrit Submission.</h1>
<p>Check out my final Pull Request here: <a href="https://review.haiku-os.org/c/haiku/+/8141">8141</a></p>
<h1 id="overview">Overview</h1>
<p>During this GSoC period, I focused on developing the virtio sound driver for Haiku,
aiming to enhance its performance as a guest OS in virtualized environments.
This journey began with some challenges, for example, initially, I missed a small
detail in the driver module path, which prevented the driver from loading.</p>
<p>One of the significant setbacks, I had, was understanding hmulti_audio.
With little to no documentation available, it felt like working with a black box.
I had to dive into existing audio drivers to piece together how things worked, but
even then, some aspects remained hidden to me.</p>
<p>Throughout the development, I encountered various bugs, such as SMAP faults and
unexpected crashes linked to hmulti_audio&rsquo;s undocumented requirements. But we
continued! And as of now, the driver supports playback but with some limitations.
High bitrates (e.g., 384 kHz, 192 kHz) are not fully functional, so playback for these
has been disabled. Recording, while partially successful at very low bitrates, has been
disabled due to quality concerns. I plan to continue refining these aspects post-GSoC,
with a focus on achieving reliable playback at high bitrates and reliable recording for
all bitrates.</p>
<h1 id="why-wont-you-play-at-normal-speed">Why won&rsquo;t you play at normal speed?</h1>
<p>VirtIO Sound relies on four queues: the control queue, the rx queue, the tx queue,
and the event queue (currently unused). Configuration and stream discovery occur
through the control queue, where we identify the supported bitrates, formats, and
other stream characteristics. Once the streams are discovered, we configure them by
selecting the highest supported bitrate, which hmulti_audio defaults to, and initiate
the buffer_exchange process. Sounds simple? It it simple, but after this&hellip; (ugh)</p>
<p>The VirtIO specification leaves room for interpretation, particularly regarding how
and when buffers are considered &ldquo;consumed.&rdquo; While the spec suggests that a device should
not return until audio buffers have been consumed, it doesn&rsquo;t clarify if this means the
buffers have been played or simply handed off to the host&rsquo;s audio system.
This ambiguity presented a challenge since hmulti_audio expects that once buffer_exchange
indicates a new buffer cycle, the buffer has been played.</p>
<p>To manage this, we implemented a solution where we wait the amount of time it would take
for the buffer to be played, aligning with hmulti_audio&rsquo;s expectations.
However, this isn&rsquo;t foolproof, as there&rsquo;s no reliable method to confirm playback—only
consumption. Events in this setup signal the completion of a full cycle, which just
results in the same behaviour. See commit <a href="https://github.com/diegoroux/haiku/commit/01f627fcd2c7994103431aa5dc3e5faf631cbe88">01f627f</a>.</p>
<h1 id="unofficial-hmulti_audio-guide">Unofficial hmulti_audio Guide</h1>
<p>I suffered through not having an idea on what should I expect to
implement and in what order. So here&rsquo;s a concise guide based on my experience:</p>
<p>In order to get loaded and recognized by hmulti_audio, the MODULE_NAME for both drivers
and devices must start with <code>drivers/audio/hmulti/{name}</code>. For example:
<code>drivers/audio/hmulti/virtio_sound/driver_v1</code> and <code>drivers/audio/hmulti/virtio_sound/device_v1</code>.
The last part signals support for the new driver version.</p>
<p>hmulti_audio communicates with drivers primarily through ioctl calls (using codes that
look like: <code>B_MULTI_...</code>). All valid ioctl codes (as well as the structs used for them) can
be found in <a href="https://github.com/haiku/haiku/blob/master/headers/private/audio/hmulti_audio.h">headers/private/audio/hmulti_audio.h</a> under the first enum.</p>
<div class="alert alert-info"><p>
For this point on, we&rsquo;ll treat sending an specific code through ioctl as calling a function.
</p>
</div>
<p>The init procedure hmulti_audio goes through for a device can be read <a href="https://github.com/haiku/haiku/blob/master/src/add-ons/media/media-add-ons/multi_audio/MultiAudioDevice.cpp#L118">here</a>.
To start, hmulti_audio retrieves the device information, such as supported formats, bitrates,
input/output channels, and the channel maps through <code>B_MULTI_GET_DESCRIPTION</code>.</p>
<div class="alert alert-info"><p>
bus_channels, can generate a bit of a confusion, at first glance, but they do not represent an
extra channel but rather the physical input/output.
</p>
</div>
<p>Secondly, hmulti_audio will try to get and set enabled channels through <code>B_MULTI_GET_ENABLED_CHANNELS</code>
and <code>B_MULTI_GET_ENABLED_CHANNELS</code>, respectively. Even if you can&rsquo;t disable/enable channels on your
device, you must reply to both calls.</p>
<p>hmulti_audio, using the information given at the <code>B_MULTI_GET_DESCRIPTION</code> step, will pick
the highest supported bitrate and format for each stream. <code>B_MULTI_SET_GLOBAL_FORMAT</code> will
be called with a <code>multi_format_info</code> argument, which will contain the requested format and
bitrate for both input and output (if both supported). Immediately after, <code>B_MULTI_GET_GLOBAL_FORMAT</code>
will be called, to obtain the formats and bitrates, on which the device is currently
operating under, this means one can ignore the initial request as <code>B_MULTI_GET_GLOBAL_FORMAT</code>
will take precedent.</p>
<p>hmulti_audio then needs to know where to send the buffers for playback or where to retrieve
recorded buffers. This is done via <code>B_MULTI_GET_BUFFERS</code>, where we provide buffer details such
as the buffer size (in frames per buffer, rather than bytes), the number of buffer cycles,
and the buffers themselves. Inside <code>multi_buffer_list</code>, an array for a set number of buffers
is given, this array (separate arrays for playback and recording exist, but are the same
in structure) allows us to specify the starting address of this buffer, for one channel,
one must give the starting point for each channel, and stride (size in bytes of a single frame)
repating this for N buffers. To illustrate this point, for one buffer:</p>
<div class="highlight"><pre tabindex="0" style="color:#d0d0d0;background-color:#202020;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#6ab825;font-weight:bold">for</span> (uint32 ch_id = <span style="color:#3677a9">0</span>; ch_id &lt; stream-&gt;channels; ch_id++) {
</span></span><span style="display:flex;"><span> buf_desc[ch_id].base = buf_ptr + (format_size * ch_id);
</span></span><span style="display:flex;"><span> buf_desc[ch_id].stride = format_size * stream-&gt;channels;
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Finally, <code>B_MULTI_LIST_MIX_CONTROLS</code> is called by hmulti_audio to query the driver
for any user-facing controls that should be displayed in the Media Preferences. It is
acceptable to return 0 controls and B_OK if none are available.</p>
<p>The call to <code>B_MULTI_BUFFER_EXCHANGE</code> signifies the start of playback/recording,
although it is possible no actual audio data is sent at this stage. If the user
changes preferences (such as the bitrate), hmulti_audio will cycle through calling
<code>B_MULTI_SET_GLOBAL_FORMAT</code> and <code>B_MULTI_GET_BUFFERS</code> again.</p>
<div class="alert alert-info"><p>
B_MULTI_BUFFER_FORCE_STOP is never called, one should never assume that
playback/recording will be explicitly stopped.
</p>
</div>
<p>All the other ioctl codes, may remain unimplemented at your discretion.</p>
<div class="alert alert-info"><p>
Remember all structs given, live in userspace. Act accordingly.
</p>
</div>
<h1 id="so-what-did-you-do-this-summer">So, what did you do this summer?</h1>
<p>I focused on implementing the VirtIO sound specification and
integrating it with Haiku&rsquo;s hmulti_audio system.</p>
<p>I successfully implemented playback, created a driver that&rsquo;s flexible and easy
to update, with potential for multi-stream support in the future.</p>
<p>This wasn&rsquo;t as straighforward as it might seem, see <a href="https://github.com/diegoroux/haiku/issues/1">#1</a>,
<a href="https://github.com/diegoroux/haiku/issues/2">#2</a>, <a href="https://github.com/diegoroux/haiku/issues/3">#3</a>.</p>
<p>Issue <a href="https://github.com/diegoroux/haiku/issues/1">#1</a>, given that I forgot
that structs came from userspace, I constantly violated SMAP in all replies
to hmulti_audio.</p>
<p>Issue <a href="https://github.com/diegoroux/haiku/issues/2">#2</a>, wanting to have &lsquo;clean&rsquo;
replies I decided to memset (0x00) some structs before writing our reply on them. Doing
that in <code>get_description</code> caused the array pointer, where the channel map was
expected to be, to be wiped to NULL. See how we <a href="https://github.com/diegoroux/haiku/commit/b3a98e80c4b5b05d331d0b4b6ad1dcc14d1e58d6">fixed this</a>.</p>
<p>Issue <a href="https://github.com/diegoroux/haiku/issues/3">#3</a>.</p>
<p><img src="../page_fault.png" alt="page fault image"></p>
<p>The longest (in time being opened to fixed) to exist. A page fault was occuring, randomly.
This made it harder to debug, as it sometimes it appeared to be fixed but one hour later
coming back. It was actually caused by not having values in the buffer_exchange for
the recording. It was fixed by a ~three line change. This was frustrating, but I&rsquo;m
glad it got fixed.</p>
<p>I dedicated a significant portion of my time to understanding and troubleshooting issues,
especially around high-bitrate playback and recording. Although high-bitrate playback is
still a work in progress, and recording has been temporarily disabled, these are areas
I plan to continue improving post-GSoC.</p>
<p>Looking ahead, my goals include fixing high-bitrate playback, re-enabling recording,
and optimizing the driver&rsquo;s performance, particularly in terms of CPU usage.
This project is something I plan to continue developing well beyond the GSoC period.</p>
<h1 id="the-end">the end?</h1>
<p>This GSoC journey has been an incredible learning experience.
I want to endlessly thank Jérôme Duval (@korli) and Scott McCreary (@scottmc) for their
invaluable guidance, patience and valuable lessons; as well as to the community, everyone who
took the time to read my blog posts, those who offered help in the IRC during the early stages.
Your support made this possible, and I look forward to continuing my work with Haiku
in the future.</p>
<p>It&rsquo;s not the end, at least not for the VirtIO Sound driver.</p>
<p>If you want to have a more detailed view of all of my work, click <a href="https://github.com/diegoroux/haiku/pull/4">here</a>.</p>
<p>Thank you all!</p>
<ul>
<li>Diego Roux</li>
</ul></description></item><item><title>[GSoC 2024] Hardware Virtualization: Final Report</title><link>https://www.haiku-os.org/blog/dalme/2024-08-19_gsoc_2024_hardware_virtualization_final_report/</link><author>dalme</author><pubDate>Wed, 21 Aug 2024 00:00:00 +0200</pubDate><guid>https://www.haiku-os.org/blog/dalme/2024-08-19_gsoc_2024_hardware_virtualization_final_report/</guid><description><h2 id="project-overview">Project overview</h2>
<p>QEMU is a virtual machine which allows running an operating system inside of another. While there already is a Haiku port, it currently does not support any acceleration system through native virtualization (through Intel VT-x and AMD SVM.) This makes it too slow for many uses.
This project aimed to bring hardware virtualization to Haiku by porting NVMM, a hypervisor that already has QEMU support, into Haiku from DragonFlyBSD. The project goals (as included on the proposal) were:</p>
<h3 id="project-goals">Project goals</h3>
<ul>
<li>NVMM driver ported to Haiku (VMX backend only)</li>
<li>QEMU capable of accelerating virtual machines through NVMM</li>
</ul>
<p>The project goals didn&rsquo;t change much during the project (although there were plans to get SVM backend working too, which didn&rsquo;t happen since I didn&rsquo;t got any time to get into it) and they&rsquo;ve been completed to some extent: The driver (VMX backend) is almost completely ported (there are only a few details missing) and QEMU is capable of accelerating virtual machines but several bugs remain.</p>
<p>Now, the details:</p>
<h2 id="completed-objectives">Completed objectives</h2>
<ul>
<li>NVMM frontend ported</li>
<li>NVMM VMX backend ported (with a couple issues remaining)</li>
<li>libnvmm (&amp; test suite) ported</li>
<li>EPT support added to the kernel</li>
<li>QEMU patched to support NVMM on Haiku and working</li>
</ul>
<h2 id="unresolved-issues">Unresolved issues</h2>
<ul>
<li>SVM backend not ported</li>
<li>Some OSes don&rsquo;t work properly on QEMU</li>
<li>Different behavior on real hardware</li>
<li>QEMU crashes when SMP is enabled</li>
<li>EPT translations not flushed from TLB on time</li>
</ul>
<p>These are just the major issues, for a full list of incomplete stuff refer to <a href="https://github.com/dalmemail/haiku-nvmm/issues">GitHub</a> (43 still opened while writing this post).
Here we can see QEMU virtualizing KolibriOS through NVMM (on nested virtualization):</p>
<p><img src="./files/blog/dalme/kolibri.png" alt="KolibriOS running on QEMU+NVMM (nested virtualization)"></p>
<h2 id="code">Code</h2>
<p>The code is available at <a href="https://github.com/dalmemail/haiku-nvmm">GitHub</a> (commits dated May 13 and later in the master branch). None of it has been merged yet since the driver isn&rsquo;t fully working
and the kernel changes I made are only needed for NVMM.</p>
<h2 id="completed-objectives-technical-details">Completed objectives: Technical details</h2>
<p>Some details about each thing I got done, in chronological order, with some of the more
relevant commits included. Some commits from the beginning of the project contain changes
to more stuff than what the commit message says since at the beginning I had to do lots
of changes in many different parts of the project and it was hard to split things into
separate commits.</p>
<h3 id="importing-missing-bsd-headers">Importing missing BSD headers</h3>
<p>While the NVMM code from DragonFlyBSD was more OS-independent than the NetBSD one it still
assumed a lot of BSD macros were available. Many of them weren&rsquo;t available on Haiku so we
brought some headers from DragonFlyBSD:</p>
<ul>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/a66d31d7ef2077fc7b81c4b4b5808226b4c3a70c">Import sys/specialreg.h from DragonFlyBSD</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/dc0c22b790e7ec466b35f98317b2b30b194d9e39">Add _IOR and similar macros to nvmm ioctl codes</a></li>
</ul>
<h3 id="making-haiku-headers-c-compatible">Making Haiku headers C compatible</h3>
<p>NVMM is written in C while huge chunks of Haiku are written on C++. We decided to keep NVMM in C (that is, compile it with gcc) and just
put C++ code into the <code>nvmm_haiku.cpp</code> file which holds any OS-dependent logic for the Haiku port, just like there are <code>nvmm_netbsd.c</code> and <code>nvmm_dragonflybsd.c</code>.
However, there are some Haiku headers we needed to include from NVMM C code. Those headers were actually supposed
to have a C-API but apparently the x86_64 versions of those headers didn&rsquo;t. I had to made them work with C:</p>
<ul>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/d0556daa6d757fd4f6b69c76ffb93e663fa2d937">Make arch/x86/arch_cpu.h kernel header C compatible</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/3769a430544ed5941c4544b0aa10c97052ab81e0">nvmm: stub driver working within whole NVMM codebase</a></li>
</ul>
<h3 id="svm-backend">SVM backend</h3>
<p>Just for the sake of completion. The only commits involving the SVM backend:</p>
<ul>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/5f9a239c34e2873b8882fa7d8d1322cdee00a4e2">nvmm: stub driver now supports AMD SVM</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/ceb67a11c93c5d24f64ba878643038de01837830">init and fini also working on the SVM backend</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/740df4501a8a5ca90652f12a04a957ece5cbc2fe">Implement os_pagemem_alloc/os_pagemem_free and modify SVM backend to avoid need of os_pa_zalloc/os_pa_free</a></li>
</ul>
<h3 id="porting-libnvmm">Porting libnvmm</h3>
<p>Porting libnvmm was straightforward and painless, although it required to remove some non-standard error codes.</p>
<ul>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/e4f51dac9bb0cea0ec6e1037c70e72e72e5df3d9">libnvmm compiles</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/b2600d524f2a5e3cc80be86bf7d206be79ddfe14">Don&rsquo;t use EPROGMISMATCH</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/406390221c026a52bc20480e58e39c9df6f26fa9">Add nvmm headers to haiku_devel package</a></li>
</ul>
<h3 id="vmx-backend">VMX backend</h3>
<p>This was the part of the project that spans greater over time (I worked on it from May 13 to around July 16, and there were bugfixes after that).
It slowly gained functionality over time, each day adding a new feature until finally <a href="https://blog.netbsd.org/tnf/entry/from_zero_to_nvmm">the calc example</a>
became the first VM to be virtualized by NVMM on Haiku.</p>
<p>The code contained here is very straight-forward since the hard part was getting the OS specific
functions that these IOCTLs needed working.</p>
<ul>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/97b04d055ab1cb984cebe3082ac710b2d1d8b46b">Getting vmx_init() to compile (not compiling yet)</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/f840ed632ca10ba52222bd7c1137c660deeb9468">Implement NVMM_IOC_CAPABILITY ioctl</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/038ac7c167a525aff7c4809ea414fb72cf6ae290">Implement NVMM_IOC_MACHINE_CREATE ioctl</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/a2588be1c3a0cd7550c3db29b7f74781eca78444">Implement NVMM_GPA_MAP ioctl</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/70fceb11ddd3019364bcbd41d580d0774954d505">Implement NVMM_IOC_VCPU_RUN ioctl</a></li>
</ul>
<p><img src="./files/blog/dalme/vm.png" alt="calc-vm virtualized by NVMM"></p>
<h3 id="haiku-specific-driver">Haiku specific driver</h3>
<p>Implemented Haiku legacy driver API into the driver.</p>
<ul>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/c081231b1ab7dc26c4d6f4297471b10a47f0335f">Driver ready to accept ioctl calls!</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/a76e570e205a1edcad147e1ca4c6ae4c348387a2">Use user_memcpy() on ioctl data forwarding from userspace</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/87f7960c478a4cc79ebe3f385dbea9f89d3ae466">Make ioctl callback thread-safe</a></li>
</ul>
<h3 id="memory-management">Memory management</h3>
<p>Arguably the hardest and longest part of the project since it was a prerequisite to implement almost everything (VMX backend and parts of NVMM frontend).
It required me to understand a big part of the virtual memory management code, how
NVMM handles memory and how QEMU maps it. Finally guest memory management is integrated into
Haiku virtual memory subsystem, as shown by the guest page fault handler which simply calls
Haiku page fault handler.</p>
<ul>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/a44217b322966d0d25b4c0f38c941577c55a6a87">Implement os_contigpa_zalloc() and os_contigpa_free()</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/d0492caaf1aac0381ef639241b788e256a31ef42">Implement os_vmobj_t create(), ref() and rel() functions</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/c83a7188245cb46a7ab2c1c2f62fc5589b6e84db">Refactor os_vmspace_t, os_vmspace_create(), os_vmspace_destroy(), os_vmspace_fault() using VMCache instead of VMAddressSpace</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/80810c681bc2dc54793dfbad66066dd253c920c3">Implement os_vmobj_unmap() and os_vmobj_rel() properly</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/80b835b446e4ed936a486e9c3ee4e8126d52f357">Implement guest page fault handler (os_vmspace_fault())</a></li>
</ul>
<h4 id="ept-extended-page-tables">EPT (Extended Page Tables)</h4>
<p>These tables handle the guest physical memory so our guest OS can access memory at nearly native speed. This meant adding <code>X86GPAtoHPATranslationMap</code> to the kernel:
a new type of translation map that handles GPA (Guest Physical Address) to HPA (Host Physical Address) translations.
<code>VMVirtualAdressSpace</code>, a new class of address space that handles the guest physical memory
address space was also added. It doesn&rsquo;t have any Haiku kernel pages mapped and uses <code>X86GPAtoHPATranslationMap</code> as translation map.
Finally we have <code>EPTPagingMethod</code> which creates, destroys and populates EPT tables. It&rsquo;s called from <code>X86GPAtoHPATranslationMap</code>.</p>
<ul>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/466be08a7c70604351f5b00118610634f935a03b">Add X86GPAtoHPATranslationMap to handle EPT tables</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/fe8e5b8c319f514b0b5ce047925cc2d09e619e0e">Add VMVirtualAddressSpace to handle guest OS address spaces</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/commit/94e2ab404893fb87dd69d8e175ce0550bee17d54">Implement EPT tables (EPTPagingMethod)</a></li>
</ul>
<h3 id="qemu">QEMU</h3>
<p>Getting QEMU to work required a new patch since QEMU buildsystem only looks for NVMM on NetBSD.
In addition to that there were a few other issues:</p>
<ul>
<li>Non-standard error codes were used.</li>
<li>QEMU couldn&rsquo;t find NVMM or libnvmm headers.</li>
</ul>
<p>Since NVMM isn&rsquo;t ready to merge yet, QEMU recipe and patch aren&rsquo;t merged either. I&rsquo;ve <a href="https://github.com/dalmemail/haiku-nvmm/commit/c478b6b56fbe3cd0f2bc30f4730d32817f31f5c6">pushed them to GitHub</a>.</p>
<ul>
<li><a href="https://github.com/dalmemail/haiku-nvmm/blob/master/qemu-8.2.2.recipe">QEMU 8.2.2 build recipe</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm/blob/master/qemu-8.2.2.patchset">QEMU 8.2.2 patchset</a></li>
</ul>
<h2 id="unresolved-issues-technical-details">Unresolved issues: Technical details</h2>
<h3 id="some-oses-dont-work-properly-on-qemu">Some OSes don&rsquo;t work properly on QEMU</h3>
<p><img src="./files/blog/dalme/QEMU1.png" alt="Haiku virtualized on Haiku"></p>
<p>As you can see it doesn&rsquo;t manage to virtualize Haiku correctly. It feels like some memory is copied wrong at some point. Xubuntu fails very early too.
Sortix appears to work fine (on nested virtualization) but that&rsquo;s likely just because it&rsquo;s a smaller OS and the live ISO loads almost everything using GRUB (which uses BIOS calls to do so). Other toy OSes appear to work fine. I never managed to get the time to debug this so I can&rsquo;t provide any more information.</p>
<h3 id="different-behavior-on-real-hardware">Different behavior on real hardware</h3>
<p>One of the biggest bugs I have debugged during GSoC consisted on very poor virtualization performance
on real hardware. When running on nested virtualization (that is, executing QEMU+NVMM inside a VM running Haiku)
things would ran slow (as expected on nested virtualization) but with similar performance as
QEMU+NVMM on NetBSD. However, when running on real hardware performance
would be way worse than that. Loading GRUB alone took a couple of minutes (if not more).
There are a couple of reasons why this was happening but I didn&rsquo;t find the &ldquo;big&rdquo; one until
yesterday. To keep it short I was convinced by some reason that EPT tables format was the same
as in X86 page tables but it&rsquo;s not (although it&rsquo;s similar enough to work). This was preventing the TLB from caching any EPT stuff and thus causing a huge performance loss.</p>
<p><a href="https://github.com/dalmemail/haiku-nvmm/commit/94e2ab404893fb87dd69d8e175ce0550bee17d54">I&rsquo;ve written the correct EPT tables into the kernel</a> and it finally runs fast. The problem (yes, there is always a new problem) is
that several new bugs have appeared. I&rsquo;ve managed to fix a few of them, but others remain:</p>
<ul>
<li>The Space Invaders boot sector game <a href="https://discuss.haiku-os.org/t/gsoc-2024-hardware-acceleration-for-haikus-qemu-port/14784/28">I showed on the forum</a> keeps working on nested virtualization but doesn&rsquo;t work anymore on real hardware. BIOS just hangs trying to load it.</li>
<li>Sortix doesn&rsquo;t work anymore. It reboots while loading GUI (probably a triple fault). It seems to work fine if you disable GUI on GRUB.</li>
</ul>
<p>Since I fixed the performance issue yesterday I just haven&rsquo;t had enough time to fix this yet.</p>
<h3 id="qemu-crashes-when-smp-is-enabled">QEMU crashes when SMP is enabled</h3>
<p>As the title says QEMU tends to get an invalid VM exit from NVMM which causes it to abort, when running on a multiprocessor machine. This happens when a VM is migrated
from one physical CPU to another, but it only happens sometimes. I prioritized fixing the performance issues and
getting QEMU+NVMM working correctly on one CPU before even trying to fix this. I took a look at the code I wrote in May to handle VM migrations and didn&rsquo;t see any obvious failure.</p>
<h3 id="ept-translations-not-flushed-from-tlb-on-time">EPT translations not flushed from TLB on time</h3>
<p>This is one of the few missing things on the VMX backend. Cached values from EPT tables (aka host TLB) needs to be flushed everytime there is a change on the EPT
tables. On NetBSD they do this by installing a callback into the translation map. On DragonFlyBSD the translation map increments a counter everytime a flush is needed. On Haiku we
don&rsquo;t yet manage this in any way. This isn&rsquo;t a problem yet because everytime we do a context switch
we flush the whole TLB (by reloading CR3) but it&rsquo;ll be a problem for having VMs with more than one CPU since we should invalidate remote TLBs too. Both the NetBSD
and the DragonFlyBSD solution should be easy to add to the brand new <code>X86GPAtoHPATranslationMap</code>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Throughout this project I&rsquo;ve learned a lot about VMX and virtualization in general. I&rsquo;ve also learned
about paging and got some hands on experience doing kernel development on a well-established project.</p>
<p>I have also learned more about assembly and debugging (GDB scripts).</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>This awesome summer wouldn&rsquo;t have been possible without my awesome mentors: waddlesplash and scottmc. Thank you!</p>
<p>I also want to thank Pulkomandy for checking my proposal draft back on March and trungnt2910 for fixing his GDB port so it would be able to debug QEMU. And of course all the people in the community that have been showing interest on this project in the forum and IRC.</p>
<p>I can&rsquo;t believe it&rsquo;s over. Time flies when you&rsquo;re having fun!</p>
<h2 id="project-links">Project links</h2>
<ul>
<li><a href="https://discuss.haiku-os.org/t/gsoc-2024-hardware-acceleration-for-haikus-qemu-port/14784/9">GSoC project thread on Forums</a></li>
<li><a href="https://github.com/dalmemail/haiku-nvmm">NVMM port code at GitHub</a></li>
</ul>
<h2 id="vmx-links">VMX links</h2>
<ul>
<li><a href="https://rayanfam.com/tutorials/">Hypervisor From Scratch</a></li>
<li><a href="https://revers.engineering/7-days-to-virtualization-a-series-on-hypervisor-development/">5 Days to Virtualization: A Series on Hypervisor Development</a></li>
<li><a href="https://revers.engineering/mmu-virtualization-via-intel-ept-index/">MMU Virtualization via Intel EPT</a></li>
<li><a href="https://www.triplefault.io/2017/07/introduction-to-ia-32e-hardware-paging.html">Introduction to IA-32e hardware paging</a></li>
</ul></description></item><item><title>[GSoC 2024] Porting WebKit2 Final Report</title><link>https://www.haiku-os.org/blog/zardshard/2024-08-16_gsoc_2024_porting_webkit2_final_report/</link><author>zardshard</author><pubDate>Tue, 20 Aug 2024 10:55:20 -0400</pubDate><guid>https://www.haiku-os.org/blog/zardshard/2024-08-16_gsoc_2024_porting_webkit2_final_report/</guid><description><p>The goal of this document is to be an overview of everything I did during GSoC. It should be readable even if you haven&rsquo;t read any of the previous blog posts and don&rsquo;t know much about Haiku or WebKit (I hope I succeeded!).</p>
<p>First, some background. Haiku&rsquo;s native browser is WebPositive.</p>
<p><img src="./files/blog/zardshard/WebPositive.png" alt="Screenshot of WebPositive"></p>
<p>WebPositive&rsquo;s code mostly deals with the user interface. It uses our fork of WebKit, HaikuWebKit, to actually render the web pages, run JavaScript, process input, and so on. WebKit (and, by extension, HaikuWebKit) provides two API versions: WebKitLegacy, and WebKit (aka WebKit2). Unfortunately, we are still stuck using WebKitLegacy even though WebKit2 has been out for more than a decade.</p>
<p>The biggest immediate advantage of switching to WebKit2 is crash resistance. It manages this by splitting the browser into several processes. Each web page runs inside of a dedicated WebProcess. If a WebProcess crashes, all of the rest of the WebProcesses should still be fine and the browser should keep running. The NetworkProcess deals with networking. Finally, the browser&rsquo;s process deals with the UI. Another advantage of switching to WebKit2 is, as far as I know, its API is more up-to-date and will allow us to do more things than WebKitLegacy did.</p>
<p>Rajagopalan worked on porting WebKit 2 to Haiku for GSoC 2019 (if you&rsquo;re interested in that project, here is the <a href="https://www.haiku-os.org/blog/rajagopalan/2019-08-15_gsoc_2019_final_report/">final report</a>, the <a href="https://github.com/haiku/webkit-old/pulls?q=is%3Apr+author%3ARAJAGOPALAN-GANGADHARAN">PRs</a>, and the <a href="https://github.com/haiku/webkit-old/commits/webkit2">commits</a>). Unfortunately, as the years have passed, the solution has bit-rotted.</p>
<p>My GSoC project for 2024 was to continue with the work on WebKit2. My first and longest task would be to get it back to a working state. I made it a goal to make the port easier to maintain so that it hopefully doesn&rsquo;t bit-rot again.</p>
<p>One final thing to know: rather than testing our implementation of WebKit2 with WebPositive, there is a very simple browser called MiniBrowser that we use that was made specifically for this purpose. Most likely, WebPositive would require several APIs that are currently not implemented. The advantage of using MiniBrowser is that it only requires a minimal set of APIs and the code for it is small enough that it can easily be updated if the API is changed.</p>
<p><img src="./files/blog/zardshard/MiniBrowser_development/3.png" alt="Screenshot of MiniBrowser. The UI is very stripped-down. It doesn&rsquo;t even have tabs!"></p>
<h1 id="what-i-did">What I did</h1>
<h2 id="1-compilation">1. Compilation</h2>
<p>The first step was getting WebKit, as well as MiniBrowser, to compile.</p>
<h3 id="the-debug-build-had-some-errors">The debug build had some errors</h3>
<p>HaikuWebKit has often been built in release-with-asserts mode. But I chose to build it in debug mode. This revealed that debug builds had bit-rotted. Fortunately, it wasn&rsquo;t severe and I only had to make a few small fixes.</p>
<p>For completeness, I&rsquo;ve included links to the PRs and commits in this report.</p>
<p>PR: <a href="https://github.com/haiku/haikuwebkit/pull/37">Fix broken debug build</a><br>
Commit: <a href="https://github.com/haiku/haikuwebkit/commit/e91178846d7ae1ee92b551eefb3453026dfd7f1c">Fix broken debug build</a></p>
<h3 id="lld---linking-doesnt-have-to-take-30-minutes">Lld - linking doesn&rsquo;t have to take 30 minutes</h3>
<p>Build times for WebKit were excruciatingly slow. One place I saw room for speeding it up was in linking. I did some experimentation with using mold for the linker, since I saw that it was the fastest linker. It took a bit of work to port it, but in the end, it worked reasonably well. But then I realized that even lld, which Haiku already had available, would be a significant improvement. Lld, unlike ld, pointed out some problematic multiple definitions. I fixed them, but by the time I was done I realized that PulkoMandy had already done it! I also discovered a <a href="https://github.com/haikuports/haikuports/issues/10445">bug</a> in our port of lld. I developed a <a href="https://github.com/haiku/haikuwebkit/pull/36">workaround</a> for the bug, but in the end, threedeyes fixed the bug very quickly and so the workaround was not necessary.</p>
<p>In one of my tests, using lld reduced linking times from 30 minutes to 1 minute! PulkoMandy added a <a href="https://github.com/haiku/haikuwebkit/tree/6cfe3dd?tab=readme-ov-file#speeding-up-the-build-with-llvm-lld">mention of lld</a> to HaikuWebKit&rsquo;s README.</p>
<h3 id="dealing-with-storage-constraints">Dealing with storage constraints</h3>
<p>I had 20 GB to use for HaikuWebKit. To make WebKit fit into that space, I had to use several tricks, including using a shallow clone and a sparse checkout. I also selectively enabled debugging information for some subfolders while keeping it disabled in others (I&rsquo;ve detailed how in a <a href="https://www.haiku-os.org/blog/zardshard/2024-05-15_building_webkit_sensibly/">blog post</a>). In the end, if I were to set up an environment to develop WebKit again in the future, I would probably use 25 GB, or more if I didn&rsquo;t want to use a sparse checkout.</p>
<h3 id="minibrowser-requires-networkprocess-and-webprocess">MiniBrowser requires NetworkProcess and WebProcess</h3>
<p>Once, I forgot to compile NetworkProcess and WebProcess along with MiniBrowser. It took a while to debug that crash. Fortunately, it was easy to tell the build system that MiniBrowser depends on WebProcess and NetworkProcess, so that they always compiled together in the future.</p>
<p>PR: <a href="https://github.com/haiku/haikuwebkit/pull/38">Compile NetworkProcess and WebProcess along with MiniBrowser</a><br>
Commit: <a href="https://github.com/haiku/haikuwebkit/commit/b544a009d3642b1c958ef1e94f3cb7408ce82810">Compile {Network|Web}Process with MiniBrowser</a></p>
<h3 id="ccache">Ccache</h3>
<p>Early on, I tried to use ccache. It occasionally crashed and if it crashed on a certain file, there was a good chance that it would always crash on that file. I couldn&rsquo;t find a good workaround, so I stopped using it.</p>
<p>Later on during GSoC, annoyed by one too many rebuilds that could have been avoided, I tried seeing whether an updated version of ccache would help. It did! No more crashes from ccache. I believe that investment of time has paid itself off several times in saved building time since, not to mention the benefit that it will have for other developers.</p>
<p>PR: <a href="https://github.com/haikuports/haikuports/pull/10616">ccache: bump version</a><br>
Commit: <a href="https://github.com/haikuports/haikuports/commit/915fb893a7f4480cf0b2b35469c8f424ccd89681">ccache: bump version (#10616)</a></p>
<h2 id="2-minibrowser-crashes-when-loading-a-web-page">2. MiniBrowser crashes when loading a web page</h2>
<p>Now, with a nice build system setup, I could finally address the bitrot!</p>
<p>So, MiniBrowser was starting up, but it would crash when trying to load a web page.</p>
<p><img src="./files/blog/zardshard/MiniBrowser_development/1.png" alt="An example of MiniBrowser crashing when navigating to a web page."></p>
<h3 id="crash-1">Crash #1</h3>
<p>The problem was a certain function needed to be implemented. So I had to make a choice: either go with an existing solution and risk needing OpenGL, or create my own implementation and risk it being difficult to create and maintain. This was a problem that required a lot of thinking on my part, but I believe I found the best solution: using the existing solution. A while later into GSoC, I found out that even though it uses OpenGL to compile, it can be made to run without actually using it!</p>
<p>My <a href="https://www.haiku-os.org/blog/zardshard/2024-05-28_gsoc_2024_fixing_the_crashing">blog post</a> on this topic and its associated discussion goes deeper into the problem and my philosophy in trying to figure out how to solve it.</p>
<div class="alert alert-info"><p>
A lot of my PRs included cleanup commits with them. I&rsquo;ve given those commits their own section, so they won&rsquo;t be listed here.
</p>
</div>
<p>PR: <a href="https://github.com/haiku/haikuwebkit/pull/39">Enable coordinated graphics</a><br>
Commits: <a href="https://github.com/haiku/haikuwebkit/commit/e611ac85a6e0c36d4ed24368ada983103ff0a616">Enable coordinated graphics</a>, <a href="https://github.com/haiku/haikuwebkit/commit/6eb9661091e840ebe5112706a8a0fd0ce9d5a550">Switch method of uniquely identifying BBitmaps</a></p>
<h3 id="crash-2-and-3">Crash #2 and #3</h3>
<p>There were two more crashes that I addressed. Both of these were easy compared to the first one. For more details, see my <a href="https://discuss.haiku-os.org/t/gsoc-2024-fixing-the-crashing-haiku-project/15054/25">posts</a> in the forum.</p>
<p>PRs: <a href="https://github.com/haiku/haikuwebkit/pull/40">Disable message attachments</a>, <a href="https://github.com/haiku/haikuwebkit/pull/41">Do not set drawing area size outside of main thread</a><br>
Commits: <a href="https://github.com/haiku/haikuwebkit/commit/9f6f0f0fcfc0586a33904b30238e8e402f6fac49">Disable message attachments</a>, <a href="https://github.com/haiku/haikuwebkit/commit/4eaac5a387e667ba3c0f8190c1f4a97db02b3436">Do not set drawing area size outside of main thread</a></p>
<h2 id="3-nothing-is-being-displayed">3. Nothing is being displayed</h2>
<p>Well, there were no more obvious crashes, but there were no web pages displaying either. MiniBrowser did not make any attempt at drawing them to the screen.</p>
<p><img src="./files/blog/zardshard/MiniBrowser_development/6.png" alt="A screenshot of a reproduction of the problem. MiniBrowser was navigated to https://www.haiku-os.org/ but made no attempt at displaying anything. The background stayed gray"></p>
<p>Why? Well, some critical functions involved in drawing weren&rsquo;t being called. But why? Well, after <a href="https://discuss.haiku-os.org/t/gsoc-2024-fixing-the-crashing-haiku-project/15054/31">going up the chain for a while</a>, I was finally able to figure out what was going on. It turns out, in some situations, the run loop was failing to process tasks given to it.</p>
<p><details><summary>What is a run loop?</summary>
The job of a RunLoop in WebKit is to receive and perform tasks. A thread with a RunLoop running will do nothing until it receives a piece of work to perform. It will then perform the work and go back to doing nothing. If multiple pieces of work arrive at the same time, RunLoop will queue the work and work on them one at a time. If this sounds like <code>BLooper</code>s, that's because they are! In fact, we use <code>BLooper</code>s to implement RunLoops on HaikuWebKit!
</details></p>
<p>The problem was that the run loop should have been instructed to start processing the first message in the queue when it is started. Otherwise, it may never start processing the messages. Ironically, I thought the problem was something else until I started writing this blog post, which made me realize that the problem was somewhat different from what I thought it was. That allowed me to improve the code further.</p>
<p>PRs: <a href="https://github.com/haiku/haikuwebkit/pull/42">Fix some IPC messages not being processed</a>, and after realizing the true cause of the problem, <a href="https://github.com/haiku/haikuwebkit/pull/52">Attach handler to looper in RunLoop::run</a><br>
Commits: <a href="https://github.com/haiku/haikuwebkit/commit/e2552027771e8eee9ca0cd67b92c8bbd1ef50045">Improve wrapping {Web|Network}Process in BApplication</a>, <a href="https://github.com/haiku/haikuwebkit/commit/83d94fe756320dce93331b95a98661c6629b81b5">Fix Haiku&rsquo;s implementation of RunLoop</a>, and after realizing the true cause of the problem, <a href="https://github.com/haiku/haikuwebkit/commit/6cfe3dd9353ad4296b1b489dfb33801f32e6c871">Attach handler to looper in RunLoop::run</a></p>
<h2 id="4-fixing-ipc">4. Fixing IPC</h2>
<p>Still, nothing was being displayed on screen, but, fortunately, unlike the previous problem, the problem was obvious. WebProcess and Network were communicating perfectly well with MiniBrowser. However, they weren&rsquo;t capable of communicating with each other.</p>
<p>Here, once again, there were multiple methods of solving the problem. Eventually I realized that it would actually be possible to use plain old UNIX sockets. Previously, PulkoMandy and I were under the impression that UNIX sockets would not work well with the way Haiku sent messages. However, it turned out that WebKit used the sockets in such a way that it was compatible, and after some <a href="https://www.freelists.org/post/haiku-development/How-to-implement-the-addition-to-Haikus-API-for-WebKits-IPC">discussion about which way to implement IPC</a>, we concluded that using UNIX sockets was the best way.</p>
<p>While I was sad that we wouldn&rsquo;t be using Haiku&rsquo;s native API, I believe now that it was the best course of action, since we wouldn&rsquo;t need to do any maintenance of IPC for WebKit in the future. It also unlocked the ability to shed more Haiku-specific code from WebKit later on.</p>
<p>This <a href="https://www.haiku-os.org/blog/zardshard/2024-06-28_gsoc_2024_fixing_ipc_in_webkit/">blog post</a> and associated discussion goes into depth on the problem and how I went about solving it.</p>
<p>PR: <a href="https://github.com/haiku/haikuwebkit/pull/43">Switch to UNIX sockets for IPC</a><br>
Commits: <a href="https://github.com/haiku/haikuwebkit/commit/40cc736d81e19810e96681e777c7a29b76952217">Fix Haiku&rsquo;s implementation of IPCSemaphore</a> - miscellaneous bugfixes, <a href="https://github.com/haiku/haikuwebkit/commit/73ffd68c24bad17a744eaa520626424e1ea45090">Switch to UNIX sockets</a></p>
<h2 id="5-fixing-shared-memory">5. Fixing shared memory</h2>
<p>As a nice side-effect of using UNIX&rsquo;s IPC, we were also able to use the existing implementation of shared memory that used UNIX&rsquo;s shared memory. Once again, it&rsquo;s a bit sad that we&rsquo;re not using Haiku&rsquo;s native API for that (except for under-the-hood, since Haiku implements UNIX&rsquo;s shared memory using its own primitives), but, hopefully, there will be no more maintenance burden for us there either!</p>
<p>The only work that was needed to make shared memory work was to make our implementation of bitmaps that could be shared across processes work regardless of which shared memory API was being used. I also did some other miscellaneous tasks at the same time, each of which brought MiniBrowser closer the displaying the web page</p>
<p>PR: <a href="https://github.com/haiku/haikuwebkit/pull/44">Make ShareableBitmapHaiku not depend on SharedMemoryHaiku</a><br>
Commits: <a href="https://github.com/haiku/haikuwebkit/commit/430d3923a48c2bae5271b01b2e523bcf3a835a46">Assume View is visible instead of invisible</a>, <a href="https://github.com/haiku/haikuwebkit/commit/e9ecc18c100b2b059b5e7f23dec2fe2341d30c1a">Shared memory should allow cloning</a>, <a href="https://github.com/haiku/haikuwebkit/commit/e2f3e0903999b239f875b706d901aa1778c22bd5">Make ShareableBitmapHaiku not depend on SharedMemoryHaiku</a>, <a href="https://github.com/haiku/haikuwebkit/commit/6afeaa5c1775a51b4d7ce9845c4a72df5212e394">Fix lifetimes of ShareableBitmap and BitmapRef</a>, <a href="https://github.com/haiku/haikuwebkit/commit/308c4dfe5873cb21853322769c11ef58e1b25688">Avoid copying bitmap data</a>.</p>
<h2 id="6-why-is-the-web-page-white">6. Why is the web page white?</h2>
<p>With a bit more work, WebKit was finally going through all of the motions of rendering something on the screen, but all it did was display a white page.</p>
<p><img src="./files/blog/zardshard/MiniBrowser_development/5.png" alt="MiniBrowser navigated to https://www.haiku-os.org/, but only displaying a white page"></p>
<p>It turned out that Haiku was clearing the bitmap! BeOS did not have that behavior nor did it make much sense for Haiku to clear the bitmap, so I submitted a <a href="https://review.haiku-os.org/c/haiku/+/7899">proposal</a> to change that unexpected behavior. While I waited for it to be accepted, I (following one of PulkoMandy&rsquo;s suggestions IIRC) stumbled on a performance improvement that also just so happened to fix the problem. However, it was pure chance that the performance improvement fixed the problem. Fortunately, my proposal was also accepted later on, so the bug should never again appear.</p>
<p>With that, web pages were rendering!</p>
<p><img src="./files/blog/zardshard/MiniBrowser_development/2.png" alt="Screenshot of review.haiku-os.org. Every letter is rendered as a square indicating an unknown glyph."></p>
<p>Well, the text obviously wasn&rsquo;t displaying right. Nor was there any way to interact with the web page.</p>
<p>A <a href="https://www.haiku-os.org/blog/zardshard/2024-07-17_gsoc_2024_drawing_to_the_screen/">blog post</a> and its discussion goes into more detail on this.</p>
<p>PR: <a href="https://github.com/haiku/haikuwebkit/pull/45">Implement displaying webpages on screen</a><br>
Commits: <a href="https://github.com/haiku/haikuwebkit/commit/1f043ddc908e29f707187013e2ae745ec97e2bad">Only use B_BITMAP_ACCEPTS_VIEWS where necessary</a> - the performance improvement, <a href="https://github.com/haiku/haikuwebkit/commit/54f6bffbff1dd68bca158744fc99571c885ad239">Implement displaying webpages on screen</a>, <a href="https://github.com/haiku/haikuwebkit/commit/ab80c8ca87e574254641d1b00c2eee755433956f">Rewrite RunLoop::TimerBase::isActive</a> - removes a deadlock, <a href="https://github.com/haiku/haikuwebkit/commit/7daf755588562cdc5f67b2e142d8972fa6d381ed">ShareableBitmap::paint: Do not lock the looper</a>, <a href="https://github.com/haiku/haikuwebkit/commit/4ef89d5b3ee862dad06cb1798620e52d17f90a84">Cleanup</a></p>
<h2 id="7-adding-mouse-support-and-fixing-text-rendering">7. Adding mouse support and fixing text rendering</h2>
<p>Both of those problems turned out to be very easy to fix. Mouse support required just a routine implementation of a function or two, and I guessed what was preventing text from rendering relatively quickly.</p>
<p>And with that, we had a relatively functional WebKit.</p>
<p><img src="./files/blog/zardshard/MiniBrowser_development/4.png" alt="Screenshot of discuss.haiku-os.org. The page is scrolled down and some text is selected. Text is being displayed properly."></p>
<p>PRs: <a href="https://github.com/haiku/haikuwebkit/pull/47">Implement mouse support</a>, <a href="https://github.com/haiku/haikuwebkit/pull/49">Always use CodePath::Simple to render fonts</a><br>
Commits: <a href="https://github.com/haiku/haikuwebkit/commit/6aad0fecc40005993d8a787a593be1bcf6026409">Implement mouse support</a>, <a href="https://github.com/haiku/haikuwebkit/commit/b48c5543feba67f2c3d2857852521e8fd3d34d48">Always use CodePath::Simple to render fonts</a></p>
<h2 id="8-never-use-opengl-egl-etc">8. Never use OpenGL, EGL, etc.</h2>
<p>A large minority of sites, such as <a href="https://discuss.haiku-os.org/">https://discuss.haiku-os.org/</a>, were still crashing MiniBrowser. Why? It turns out, accelerated compositing wasn&rsquo;t being disabled properly. So if WebKit thought a site might perform better with accelerated compositing, it enabled it. Well, EGL doesn&rsquo;t work properly on Haiku, and there were things to be implemented in WebKit even if EGL was around. Needless to say, WebKit would crash. Fortunately, after <a href="https://www.haiku-os.org/blog/zardshard/2024-08-13_gsoc_2024_can_i_get_discusshaiku-osorg_to_work/">considering</a> several ways of fixing it, I found an elegant solution for the problem as well.</p>
<p>I also found a crash while scrolling, and an assert that was failing for some reason (details are in the commit).</p>
<p>PR: <a href="https://github.com/haiku/haikuwebkit/pull/51">Fix crashes that occur when visiting discuss.haiku-os.org</a><br>
Commits: <a href="https://github.com/haiku/haikuwebkit/commit/991489cf11a8cc75c8cae0c82840cf3d47b73878">Fix disabling of accelerated compositing mode</a>, <a href="https://github.com/haiku/haikuwebkit/commit/a4f91475f48c7501f0d4bd13b4227fa02f0e7222">Fix crash when scrolling on some sites</a>, <a href="https://github.com/haiku/haikuwebkit/commit/e18c9c56ffcce94988f7eeac1acd950b5b26237d">Comment out failing assert</a></p>
<h2 id="9-cleanup">9. Cleanup</h2>
<p>While making changes, I often found places in the code that could be cleaned up. Often, I included these improvements along with the PR of the other changes that I was making at the time. Some got their own PR.</p>
<p>Changes with their own PR: <a href="https://github.com/haiku/haikuwebkit/pull/46">Cleanup</a>, <a href="https://github.com/haiku/haikuwebkit/pull/48">Make WebView use C++ API instead of C API</a><br>
Commits: <a href="https://github.com/haiku/haikuwebkit/commit/75e1866e8f81d9bb3355a4013deb5864c795292b">Remove unused code</a>, <a href="https://github.com/haiku/haikuwebkit/commit/7c3448810ce74a44ffccda3289bb6b34f5b52ccc">Cleanup WebViewBase</a>, <a href="https://github.com/haiku/haikuwebkit/commit/6f58a434bb62564a1630aaf2672c9ae0544d7ce7">Cleanup</a>, <a href="https://github.com/haiku/haikuwebkit/commit/7029c6b23e7efcb628febfafe0221d8f3190b679">Remove WebView::initializeOnce</a>, <a href="https://github.com/haiku/haikuwebkit/commit/ad034fa52f973a0549ecd1c41ccd2a6c7276b774">Mark some functions as unimplemented</a>, <a href="https://github.com/haiku/haikuwebkit/commit/884309b0f0c37ea297c53a855f7c2cdbaa5e4701">Remove Haiku&rsquo;s shared memory implementation</a>, <a href="https://github.com/haiku/haikuwebkit/commit/74feccc6fb7e946c114736b4d7b84543f434ed7c">Finish removing Haiku&rsquo;s shared memory implementation</a>, <a href="https://github.com/haiku/haikuwebkit/commit/9ef73271991f1735dd498d6b426f51d8205ad070">Cleanup</a>, <a href="https://github.com/haiku/haikuwebkit/commit/2def012cb79f18c886136fa78914f522b12cf310">Rewrite MiniBrowser&rsquo;s CMakeLists</a>, <a href="https://github.com/haiku/haikuwebkit/commit/2477e9fa22d3e2c8e411f42134bcaf63942e193f">Make WebView use C++ API instead of C API</a>, <a href="https://github.com/haiku/haikuwebkit/commit/7fe2fc3bf508a75da93e4c82d1babab4004ac81d">Send updated value along with DID_CHANGE messages</a>, <a href="https://github.com/haiku/haikuwebkit/commit/6eb67aa3be9f6c67c0a0a529644d024bd153b313">Cleanup</a>, <a href="https://github.com/haiku/haikuwebkit/commit/0d4b3f0d5d53fe260dd7bb67885451cd05d0fe3f">Cleanup</a></p>
<h1 id="conclusion">Conclusion</h1>
<p>I&rsquo;m somewhat surprised that I wasn&rsquo;t able to get further during GSoC. I thought at the beginning, that I had a chance of getting significantly further with my project because of the work done from the previous GSoC project. Reviewing what I&rsquo;ve done, I think I know why. First and most obvious, I had to deal with bitrot. I had to spend a lot of time debugging things that should have been working but weren&rsquo;t. In the case of messages being stuck in the run loop, it took a week or two to figure out what the problem was. It doesn&rsquo;t help that our debugger is slow and buggy. Hopefully gdb will work well as a replacement in the future! It may have taken me less time had the run loop needed to be written from scratch in the first place, since at least then exactly what the problem was would have been obvious! Secondly, I revised and reimplemented several things. For example, I had to go through the whole implementation process again for IPC before I could realize that using UNIX sockets was a good idea. Finally, it took time to figure out how to do things well so that they would not be brittle or hard to maintain.</p>
<p>Anyway, I&rsquo;ve intentionally prioritized the hard problems during GSoC, so hopefully most of what remains are the easier ones. Here&rsquo;s a list of things that still need to be done:</p>
<ul>
<li>MiniBrowser still has plenty of crashes
<ul>
<li>When closed after a web page has loaded</li>
<li>When being resized</li>
<li>When going backwards in history</li>
</ul>
</li>
<li>There are still some things that need to be implemented
<ul>
<li>Mouse scrolling doesn&rsquo;t work</li>
<li>There is no keyboard support</li>
<li>Context menus don&rsquo;t work</li>
<li>Copy-paste doesn&rsquo;t work</li>
<li>And probably more</li>
</ul>
</li>
<li>There are some things that would be nice to have
<ul>
<li>WebProcess and NetworkProcess shouldn&rsquo;t appear in the list of open applications; only the browser itself should.</li>
<li>Performance improvements - currently, WebKit2 is slightly slower than WebKitLegacy on most web pages. It is significantly slower on sites like youtube.com.</li>
<li>Videos &ndash; Alas, switching to WebKit2 did not fix video playback on youtube.com. Most likely there is an API that we need to implement that isn&rsquo;t implemented.</li>
<li>Make the tests pass &ndash; WebKit has tests available, but they&rsquo;re excluded in my sparse checkout, and, from what I&rsquo;ve heard, they don&rsquo;t work on Haiku at present. The tests will probably tell us many other areas of improvement.</li>
</ul>
</li>
<li>Our WebKit2 API for applications to use needs to be improved. The API that currently exists is a bit messy.</li>
<li>And last but not least, WebKit2 needs to be integrated with WebPositive.</li>
</ul>
<p>For further reading into my project, my blog posts and their associated discussions detail my progress pretty well.</p>
<p>Thank you to Waddlesplash, PulkoMandy, and various others who offered suggestions, reviewed code, and just in general, made the community fun to be around. And definitely, a big thank you to Google Summer of Code. Without them, the project would not have been possible for me in the first place!</p></description></item><item><title>[GSoC 2024] Improving the Userland Debugging Experience - Final Report</title><link>https://www.haiku-os.org/blog/trungnt2910/2024-08-18_gsoc_2024_debugging_final_report/</link><author>trungnt2910</author><pubDate>Sun, 18 Aug 2024 00:00:00 +0000</pubDate><guid>https://www.haiku-os.org/blog/trungnt2910/2024-08-18_gsoc_2024_debugging_final_report/</guid><description><h1 id="project-overview">Project overview</h1>
<p>A part of Google Summer of Code 2024, this project aimed to improve the userland debugging
experience for Haiku app developers, boosting the process of building and porting complex
applications.</p>
<p>The first objective was to have a working build of a modern version of GDB running on Haiku
<code>x86_64</code> - the most popular architecture with stable Haiku. Using some ideas from the incomplete
<a href="https://github.com/haikuports/haikuports/commit/985c28d7c71c911af2bf0b8b5984f32d5751fafc">recipe</a>
for GDB 8.1, I have ported GDB 15.1 to Haiku from the ground up.</p>
<p>During this process, I have detailed my implementation choices and some relevant internal parts of
both GDB and the Haiku debugger API through my
<a href="https://www.haiku-os.org/blog/trungnt2910/">blog posts</a>.</p>
<p>GDB 15.1 is now available on HaikuPorts, and can be installed using <code>pkgman install gdb</code>.</p>
<p>The original proposal also mentioned fixing the built-in <code>Debugger</code> and porting another popular
debugger, LLDB. However, multiple problems encountered when dealing with subtleties in the GDB
codebase have prevented this from happening in the project&rsquo;s timeline. I chose to prioritize
ensuring the quality of the GDB port, providing a stable tool for developers to connect to their
favorite IDEs, both on Haiku (Qt Creator) or other OSes (NetBeans, VS Code). This, I believe, should
deliever the greatest impact to the developer community.</p>
<h2 id="completed-objectives">Completed objectives</h2>
<p>I have built a stable version of GDB on Haiku:</p>
<ul>
<li>Had the latest GDB (15.1, instead of the then current 14.1) running with applicable local
debugging features (launching, attaching, breakpoints, memory, registers, and OS information)
working on <code>x86_64</code> Haiku.</li>
<li>Ensured the port&rsquo;s compatibility with popular IDE frontends.</li>
<li>Tested GDB&rsquo;s ability to debug typical Haiku programs, such as <code>Debugger</code> itself.</li>
<li>Distributed the port as a HaikuPorts package.</li>
</ul>
<h2 id="unresolved-issues">Unresolved issues</h2>
<p>The project did not have enough time to move on to the next phases:</p>
<ul>
<li><code>Debugger</code>: While work has been done to ensure GDB can consistently attach to <code>Debugger</code>, no
issues have been worked on.</li>
<li>LLDB: The project ended before work is done on LLDB. However, the information in my blog posts
would greatly assist future LLDB porting attempts:
<ul>
<li><code>Plugin/Platforms</code> corresponds to GDB&rsquo;s target-dependent <code>tdep</code> code.</li>
<li><code>Plugin/Process</code> corresponds to GDB&rsquo;s native-dependent <code>nat</code> code. <code>NativeProcess</code> is similar
to my port&rsquo;s <code>team_debug_context</code>, and <code>NativeThread</code> is similar to <code>thread_debug_context</code>.</li>
</ul>
</li>
</ul>
<h2 id="future-work">Future work</h2>
<p>These are the work I expect to do in the near future after GSoC ends:</p>
<ul>
<li>Work with GDB developers to upstream the rest of the patches.</li>
<li>Port GDB to other architectures with high interest, such as <code>x86</code> or <code>riscv</code>.</li>
<li>Respond to any community requests through GitHub issues and the discussion forum.</li>
</ul>
<p>I can also help with future attempts to port LLDB or to bring more GDB features to Haiku such as
automatically fetching debug information, if there is more community interest.</p>
<h1 id="technical-details">Technical details</h1>
<h2 id="project-structure">Project structure</h2>
<h3 id="gdb-fork">GDB fork</h3>
<p>For the project, I work on a fork of <code>binutils-gdb</code> on
<a href="https://github.com/trungnt2910/gdb-haiku">GitHub</a>.</p>
<p>The port is based on one of GDB&rsquo;s release branches, <code>gdb-{version}-branch</code>. The corresponding Haiku
branches are named <code>gdb-{version}-haiku</code>, including:</p>
<ul>
<li><code>gdb-8.1-haiku</code>: Early attempts to fix the old patchset, now abandoned.</li>
<li><code>gdb-14-haiku</code>: My first functional port of GDB. Now abandoned after the release of GDB 15.</li>
<li><code>gdb-15-haiku</code>: My latest port with all the features and bugfixes during this project.</li>
</ul>
<p>The commits are squashed to match what will be sent to HaikuPorts and upstream to GDB and do not
reflect actual development progress.</p>
<h3 id="haikuports-recipe">HaikuPorts recipe</h3>
<p>The correct commands to build the GDB port are located in the
<a href="https://github.com/haikuports/haikuports/blob/master/dev-util/gdb/gdb-15.1.recipe">HaikuPorts</a>
recipe, <code>dev-util/gdb/gdb-15.1.recipe</code>. A patchset mirroring the <code>gdb-15-haiku</code> branch is also
included.</p>
<h2 id="list-of-contributions-during-gsoc">List of contributions during GSoC</h2>
<h3 id="haikuhaiku">haiku/haiku</h3>
<p>Changes were made to the kernel to expose more debugging functionality or reasonably change some
behavior to match what GDB expects.</p>
<ul>
<li><a href="https://review.haiku-os.org/c/haiku/+/7642">headers/bsd: Remove Haiku-specific include deps (#7642)</a></li>
<li><a href="https://review.haiku-os.org/c/haiku/+/7736">kernel/debug: Report team/thread exit status (#7736)</a></li>
<li><a href="https://review.haiku-os.org/c/haiku/+/7756">kernel/debug: Report killing signals (#7756)</a></li>
<li><a href="https://review.haiku-os.org/c/haiku/+/7796">kernel/signal: Make DEBUG_THREAD non-deferrable (#7796)</a></li>
<li><a href="https://review.haiku-os.org/c/haiku/+/7797">kernel/debug: Report new team before resuming it (#7797)</a></li>
</ul>
<h3 id="haikuportshaikuports">haikuports/haikuports</h3>
<p>This contains commands and source changes required to build the GDB port on Haiku.
A browsable version of the codebase is available on
<a href="https://github.com/trungnt2910/gdb-haiku/tree/019ad2f8cffccb9ab8608034455165135d2e116c">my GitHub repository</a></p>
<ul>
<li><a href="https://github.com/haikuports/haikuports/pull/10727">gdb: Add recipe for GDB 15.1 (#10727)</a></li>
<li><a href="https://github.com/haikuports/haikuports/pull/10799">gdb: Improve GDB 15 stability (#10799)</a></li>
</ul>
<h3 id="binutils-gdb">binutils-gdb</h3>
<p>This patch is currently under review by GDB maintainers for upstreaming.</p>
<ul>
<li><a href="https://sourceware.org/pipermail/gdb-patches/2024-July/210771.html">[PATCH v2][PR 30630] gdb/remote: fix assertion failure during startup</a></li>
</ul>
<h1 id="conclusion">Conclusion</h1>
<p>That&rsquo;s the end of another Google Summer of Code season!</p>
<p>I hope the Haiku community finds my contributions useful, and I look forward to deliver more
interesting projects in the future as the OS reaches R1.</p>
<h1 id="acknowledgements">Acknowledgements</h1>
<p>First of all, I would like to thank my mentor, waddlesplash, for his help in demystifying the deep
corners of the OS and discussing solutions to more complex problems.</p>
<p>I would like to thank the Haiku organization for guiding me through the open source journey since
the days I started programming.</p>
<p>I appreciate all the community members who have tested my work and reported issues, especially X512
and my fellow GSoC participants dalme and zardshard.</p>
<p>And once again, I would like to thank my family for all the valuable support for my application and
participation in this program.</p></description></item></channel></rss>