public inbox for git-commits@fedoraproject.org
help / color / mirror / Atom feed
From: Kevin Buettner <kevinb@redhat.com>
To: git-commits@fedoraproject.org
Subject: [rpms/gdb] gdb-17.2-rebase-f44: Backport upstream patches which fix multi-threaded debugging for
Date: Sun, 28 Jun 2026 00:00:19 GMT	[thread overview]
Message-ID: <178260481954.1.12383930497775734785.rpms-gdb-71264766c54f@fedoraproject.org> (raw)

          A new commit has been pushed.

          Repo   : rpms/gdb
          Branch : gdb-17.2-rebase-f44
          Commit : 71264766c54f593fe5a98059a3492f5d8c95a600
          Author : Kevin Buettner <kevinb@redhat.com>
          Date   : 2021-06-14T23:24:14-07:00
          Stats  : +738/-1 in 9 file(s)
          URL    : https://src.fedoraproject.org/rpms/gdb/c/71264766c54f593fe5a98059a3492f5d8c95a600?branch=gdb-17.2-rebase-f44

          Log:
          Backport upstream patches which fix multi-threaded debugging for
glibc-2.34 (RHBZ 1971096, Simon Marchi, Kevin Buettner).

---
diff --git a/_gdb.spec.Patch.include b/_gdb.spec.Patch.include
index 9f233e0..219586e 100644
--- a/_gdb.spec.Patch.include
+++ b/_gdb.spec.Patch.include
@@ -384,3 +384,22 @@ Patch092: gdb-gdb27743-psymtab-imported-unit.patch
 # (Tom de Vries)
 Patch093: gdb-dont-overwrite-fsgsbase-m32.patch
 
+# Backport patch fixing gdb on glibc-2.34 machines with regard to attaching
+# to processes (RH BZ 1971096).
+Patch094: gdb-rhbz1971096-glibc2.34-1.patch
+
+# Backport patch fixing gdb on glibc-2.34 machines w/ regard to
+# libthread_db initialization.
+Patch095: gdb-rhbz1971096-glibc2.34-2.patch
+
+# Backport testsuite patch for matching new libthread_db related output
+# when testing gdb on glibc-2.34 matchines (RH BZ 1971096).
+Patch096: gdb-rhbz1971096-glibc2.34-3.patch
+
+# Backport patch adjusting test gdb.base/print-symbol-loading.exp.
+# (RH BZ 1971096).
+Patch097: gdb-rhbz1971096-glibc2.34-4.patch
+
+# Backport patch adjusting test gdb.mi/mi-sym-info.exp (RH BZ 1971096).
+Patch098: gdb-rhbz1971096-glibc2.34-5.patch
+

diff --git a/_gdb.spec.patch.include b/_gdb.spec.patch.include
index 32f5b04..b089bb1 100644
--- a/_gdb.spec.patch.include
+++ b/_gdb.spec.patch.include
@@ -91,3 +91,8 @@
 %patch091 -p1
 %patch092 -p1
 %patch093 -p1
+%patch094 -p1
+%patch095 -p1
+%patch096 -p1
+%patch097 -p1
+%patch098 -p1

diff --git a/_patch_order b/_patch_order
index 562074d..a767691 100644
--- a/_patch_order
+++ b/_patch_order
@@ -91,3 +91,8 @@ gdb-rhbz1964167-fortran-fix-type-format-mismatch-in-f-lang.c.patch
 gdb-rhbz1898252-loadable-section-outside-ELF-segments.patch
 gdb-gdb27743-psymtab-imported-unit.patch
 gdb-dont-overwrite-fsgsbase-m32.patch
+gdb-rhbz1971096-glibc2.34-1.patch
+gdb-rhbz1971096-glibc2.34-2.patch
+gdb-rhbz1971096-glibc2.34-3.patch
+gdb-rhbz1971096-glibc2.34-4.patch
+gdb-rhbz1971096-glibc2.34-5.patch

diff --git a/gdb-rhbz1971096-glibc2.34-1.patch b/gdb-rhbz1971096-glibc2.34-1.patch
new file mode 100644
index 0000000..6882699
--- /dev/null
+++ b/gdb-rhbz1971096-glibc2.34-1.patch
@@ -0,0 +1,307 @@
+From FEDORA_PATCHES Mon Sep 17 00:00:00 2001
+From: Kevin Buettner <kevinb@redhat.com>
+Date: Mon, 14 Jun 2021 19:12:06 -0700
+Subject: gdb-rhbz1971096-glibc2.34-1.patch
+
+;; Backport patch fixing gdb on glibc-2.34 machines with regard to attaching
+;; to processes (RH BZ 1971096).
+
+gdb: try to load libthread_db only after reading all shared libraries when attaching / handling a fork child
+
+When trying to attach to a pthread process on a Linux system with glibc 2.33,
+we get:
+
+    $ ./gdb -q -nx --data-directory=data-directory -p 1472010
+    Attaching to process 1472010
+    [New LWP 1472013]
+    [New LWP 1472014]
+    [New LWP 1472015]
+    Error while reading shared library symbols for /usr/lib/libpthread.so.0:
+    Cannot find user-level thread for LWP 1472015: generic error
+    0x00007ffff6d3637f in poll () from /usr/lib/libc.so.6
+    (gdb)
+
+When attaching to a process (or handling a fork child, an operation very
+similar to attaching), GDB reads the shared library list from the
+process.  For each shared library (if "set auto-solib-add" is on), it
+reads its symbols and calls the "new_objfile" observable.
+
+The libthread-db code monitors this observable, and if it sees an
+objfile named somewhat like "libpthread.so" go by, it tries to load
+libthread_db.so in the GDB process itself.  libthread_db knows how to
+navigate libpthread's data structures to get information about the
+existing threads.
+
+To locate these data structures, libthread_db calls ps_pglobal_lookup
+(implemented in proc-service.c), passing in a symbol name and expecting
+an address in return.
+
+Before glibc 2.33, libthread_db always asked for symbols found in
+libpthread.  There was no ordering problem: since we were always trying
+to load libthread_db in reaction to processing libpthread (and reading
+in its symbols) and libthread_db only asked symbols from libpthread, the
+requested symbols could always be found.  Starting with glibc 2.33,
+libthread_db now asks for a symbol name that can be found in
+/lib/ld-linux-x86-64.so.2 (_rtld_global).  And the ordering in which GDB
+reads the shared libraries from the inferior when attaching is
+unfortunate, in that libpthread is processed before ld-linux.  So when
+loading libthread_db in reaction to processing libpthread, and
+libthread_db requests the symbol that is from ld-linux, GDB is not yet
+able to supply it.
+
+That problematic symbol lookup happens in the thread_from_lwp function,
+when we call td_ta_map_lwp2thr_p, and an exception is thrown at this
+point:
+
+    #0  0x00007ffff6681012 in __cxxabiv1::__cxa_throw (obj=0x60e000006100, tinfo=0x555560033b50 <typeinfo for gdb_exception_error>, dest=0x55555d9404bc <gdb_exception_error::~gdb_exception_error()>) at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_throw.cc:78
+    #1  0x000055555e5d3734 in throw_it(return_reason, errors, const char *, typedef __va_list_tag __va_list_tag *) (reason=RETURN_ERROR, error=GENERIC_ERROR, fmt=0x55555f0c5360 "Cannot find user-level thread for LWP %ld: %s", ap=0x7fffffffaae0) at /home/simark/src/binutils-gdb/gdbsupport/common-exceptions.cc:200
+    #2  0x000055555e5d37d4 in throw_verror (error=GENERIC_ERROR, fmt=0x55555f0c5360 "Cannot find user-level thread for LWP %ld: %s", ap=0x7fffffffaae0) at /home/simark/src/binutils-gdb/gdbsupport/common-exceptions.cc:208
+    #3  0x000055555e0b0ed2 in verror (string=0x55555f0c5360 "Cannot find user-level thread for LWP %ld: %s", args=0x7fffffffaae0) at /home/simark/src/binutils-gdb/gdb/utils.c:171
+    #4  0x000055555e5e898a in error (fmt=0x55555f0c5360 "Cannot find user-level thread for LWP %ld: %s") at /home/simark/src/binutils-gdb/gdbsupport/errors.cc:43
+    #5  0x000055555d06b4bc in thread_from_lwp (stopped=0x617000035d80, ptid=...) at /home/simark/src/binutils-gdb/gdb/linux-thread-db.c:418
+    #6  0x000055555d07040d in try_thread_db_load_1 (info=0x60c000011140) at /home/simark/src/binutils-gdb/gdb/linux-thread-db.c:912
+    #7  0x000055555d071103 in try_thread_db_load (library=0x55555f0c62a0 "libthread_db.so.1", check_auto_load_safe=false) at /home/simark/src/binutils-gdb/gdb/linux-thread-db.c:1014
+    #8  0x000055555d072168 in try_thread_db_load_from_sdir () at /home/simark/src/binutils-gdb/gdb/linux-thread-db.c:1091
+    #9  0x000055555d072d1c in thread_db_load_search () at /home/simark/src/binutils-gdb/gdb/linux-thread-db.c:1146
+    #10 0x000055555d07365c in thread_db_load () at /home/simark/src/binutils-gdb/gdb/linux-thread-db.c:1203
+    #11 0x000055555d07373e in check_for_thread_db () at /home/simark/src/binutils-gdb/gdb/linux-thread-db.c:1246
+    #12 0x000055555d0738ab in thread_db_new_objfile (objfile=0x61300000c0c0) at /home/simark/src/binutils-gdb/gdb/linux-thread-db.c:1275
+    #13 0x000055555bd10740 in std::__invoke_impl<void, void (*&)(objfile*), objfile*> (__f=@0x616000068d88: 0x55555d073745 <thread_db_new_objfile(objfile*)>) at /usr/include/c++/10.2.0/bits/invoke.h:60
+    #14 0x000055555bd02096 in std::__invoke_r<void, void (*&)(objfile*), objfile*> (__fn=@0x616000068d88: 0x55555d073745 <thread_db_new_objfile(objfile*)>) at /usr/include/c++/10.2.0/bits/invoke.h:153
+    #15 0x000055555bce0392 in std::_Function_handler<void (objfile*), void (*)(objfile*)>::_M_invoke(std::_Any_data const&, objfile*&&) (__functor=..., __args#0=@0x7fffffffb4a0: 0x61300000c0c0) at /usr/include/c++/10.2.0/bits/std_function.h:291
+    #16 0x000055555d3595c0 in std::function<void (objfile*)>::operator()(objfile*) const (this=0x616000068d88, __args#0=0x61300000c0c0) at /usr/include/c++/10.2.0/bits/std_function.h:622
+    #17 0x000055555d356b7f in gdb::observers::observable<objfile*>::notify (this=0x555566727020 <gdb::observers::new_objfile>, args#0=0x61300000c0c0) at /home/simark/src/binutils-gdb/gdb/../gdbsupport/observable.h:106
+    #18 0x000055555da3f228 in symbol_file_add_with_addrs (abfd=0x61200001ccc0, name=0x6190000d9090 "/usr/lib/libpthread.so.0", add_flags=..., addrs=0x7fffffffbc10, flags=..., parent=0x0) at /home/simark/src/binutils-gdb/gdb/symfile.c:1131
+    #19 0x000055555da3f763 in symbol_file_add_from_bfd (abfd=0x61200001ccc0, name=0x6190000d9090 "/usr/lib/libpthread.so.0", add_flags=<error reading variable: Cannot access memory at address 0xffffffffffffffb0>, addrs=0x7fffffffbc10, flags=<error reading variable: Cannot access memory at address 0xffffffffffffffc0>, parent=0x0) at /home/simark/src/binutils-gdb/gdb/symfile.c:1167
+    #20 0x000055555d95f9fa in solib_read_symbols (so=0x6190000d8e80, flags=...) at /home/simark/src/binutils-gdb/gdb/solib.c:681
+    #21 0x000055555d96233d in solib_add (pattern=0x0, from_tty=0, readsyms=1) at /home/simark/src/binutils-gdb/gdb/solib.c:987
+    #22 0x000055555d93646e in enable_break (info=0x608000008f20, from_tty=0) at /home/simark/src/binutils-gdb/gdb/solib-svr4.c:2238
+    #23 0x000055555d93cfc0 in svr4_solib_create_inferior_hook (from_tty=0) at /home/simark/src/binutils-gdb/gdb/solib-svr4.c:3049
+    #24 0x000055555d96610d in solib_create_inferior_hook (from_tty=0) at /home/simark/src/binutils-gdb/gdb/solib.c:1195
+    #25 0x000055555cdee318 in post_create_inferior (from_tty=0) at /home/simark/src/binutils-gdb/gdb/infcmd.c:318
+    #26 0x000055555ce00e6e in setup_inferior (from_tty=0) at /home/simark/src/binutils-gdb/gdb/infcmd.c:2439
+    #27 0x000055555ce59c34 in handle_one (event=...) at /home/simark/src/binutils-gdb/gdb/infrun.c:4887
+    #28 0x000055555ce5cd00 in stop_all_threads () at /home/simark/src/binutils-gdb/gdb/infrun.c:5064
+    #29 0x000055555ce7f0da in stop_waiting (ecs=0x7fffffffd170) at /home/simark/src/binutils-gdb/gdb/infrun.c:8006
+    #30 0x000055555ce67f5c in handle_signal_stop (ecs=0x7fffffffd170) at /home/simark/src/binutils-gdb/gdb/infrun.c:6062
+    #31 0x000055555ce63653 in handle_inferior_event (ecs=0x7fffffffd170) at /home/simark/src/binutils-gdb/gdb/infrun.c:5727
+    #32 0x000055555ce4f297 in fetch_inferior_event () at /home/simark/src/binutils-gdb/gdb/infrun.c:4105
+    #33 0x000055555cdbe3bf in inferior_event_handler (event_type=INF_REG_EVENT) at /home/simark/src/binutils-gdb/gdb/inf-loop.c:42
+    #34 0x000055555d018047 in handle_target_event (error=0, client_data=0x0) at /home/simark/src/binutils-gdb/gdb/linux-nat.c:4060
+    #35 0x000055555e5ea77e in handle_file_event (file_ptr=0x60600008b1c0, ready_mask=1) at /home/simark/src/binutils-gdb/gdbsupport/event-loop.cc:575
+    #36 0x000055555e5eb09c in gdb_wait_for_event (block=0) at /home/simark/src/binutils-gdb/gdbsupport/event-loop.cc:701
+    #37 0x000055555e5e8d19 in gdb_do_one_event () at /home/simark/src/binutils-gdb/gdbsupport/event-loop.cc:212
+    #38 0x000055555dd6e0d4 in wait_sync_command_done () at /home/simark/src/binutils-gdb/gdb/top.c:528
+    #39 0x000055555dd6e372 in maybe_wait_sync_command_done (was_sync=0) at /home/simark/src/binutils-gdb/gdb/top.c:545
+    #40 0x000055555d0ec7c8 in catch_command_errors (command=0x55555ce01bb8 <attach_command(char const*, int)>, arg=0x7fffffffe28d "1472010", from_tty=1, do_bp_actions=false) at /home/simark/src/binutils-gdb/gdb/main.c:452
+    #41 0x000055555d0f03ad in captured_main_1 (context=0x7fffffffdd10) at /home/simark/src/binutils-gdb/gdb/main.c:1149
+    #42 0x000055555d0f1239 in captured_main (data=0x7fffffffdd10) at /home/simark/src/binutils-gdb/gdb/main.c:1232
+    #43 0x000055555d0f1315 in gdb_main (args=0x7fffffffdd10) at /home/simark/src/binutils-gdb/gdb/main.c:1257
+    #44 0x000055555bb70cf9 in main (argc=7, argv=0x7fffffffde88) at /home/simark/src/binutils-gdb/gdb/gdb.c:32
+
+The exception is caught here:
+
+    #0  __cxxabiv1::__cxa_begin_catch (exc_obj_in=0x60e0000060e0) at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_catch.cc:84
+    #1  0x000055555d95fded in solib_read_symbols (so=0x6190000d8e80, flags=...) at /home/simark/src/binutils-gdb/gdb/solib.c:689
+    #2  0x000055555d96233d in solib_add (pattern=0x0, from_tty=0, readsyms=1) at /home/simark/src/binutils-gdb/gdb/solib.c:987
+    #3  0x000055555d93646e in enable_break (info=0x608000008f20, from_tty=0) at /home/simark/src/binutils-gdb/gdb/solib-svr4.c:2238
+    #4  0x000055555d93cfc0 in svr4_solib_create_inferior_hook (from_tty=0) at /home/simark/src/binutils-gdb/gdb/solib-svr4.c:3049
+    #5  0x000055555d96610d in solib_create_inferior_hook (from_tty=0) at /home/simark/src/binutils-gdb/gdb/solib.c:1195
+    #6  0x000055555cdee318 in post_create_inferior (from_tty=0) at /home/simark/src/binutils-gdb/gdb/infcmd.c:318
+    #7  0x000055555ce00e6e in setup_inferior (from_tty=0) at /home/simark/src/binutils-gdb/gdb/infcmd.c:2439
+    #8  0x000055555ce59c34 in handle_one (event=...) at /home/simark/src/binutils-gdb/gdb/infrun.c:4887
+    #9  0x000055555ce5cd00 in stop_all_threads () at /home/simark/src/binutils-gdb/gdb/infrun.c:5064
+    #10 0x000055555ce7f0da in stop_waiting (ecs=0x7fffffffd170) at /home/simark/src/binutils-gdb/gdb/infrun.c:8006
+    #11 0x000055555ce67f5c in handle_signal_stop (ecs=0x7fffffffd170) at /home/simark/src/binutils-gdb/gdb/infrun.c:6062
+    #12 0x000055555ce63653 in handle_inferior_event (ecs=0x7fffffffd170) at /home/simark/src/binutils-gdb/gdb/infrun.c:5727
+    #13 0x000055555ce4f297 in fetch_inferior_event () at /home/simark/src/binutils-gdb/gdb/infrun.c:4105
+    #14 0x000055555cdbe3bf in inferior_event_handler (event_type=INF_REG_EVENT) at /home/simark/src/binutils-gdb/gdb/inf-loop.c:42
+    #15 0x000055555d018047 in handle_target_event (error=0, client_data=0x0) at /home/simark/src/binutils-gdb/gdb/linux-nat.c:4060
+    #16 0x000055555e5ea77e in handle_file_event (file_ptr=0x60600008b1c0, ready_mask=1) at /home/simark/src/binutils-gdb/gdbsupport/event-loop.cc:575
+    #17 0x000055555e5eb09c in gdb_wait_for_event (block=0) at /home/simark/src/binutils-gdb/gdbsupport/event-loop.cc:701
+    #18 0x000055555e5e8d19 in gdb_do_one_event () at /home/simark/src/binutils-gdb/gdbsupport/event-loop.cc:212
+    #19 0x000055555dd6e0d4 in wait_sync_command_done () at /home/simark/src/binutils-gdb/gdb/top.c:528
+    #20 0x000055555dd6e372 in maybe_wait_sync_command_done (was_sync=0) at /home/simark/src/binutils-gdb/gdb/top.c:545
+    #21 0x000055555d0ec7c8 in catch_command_errors (command=0x55555ce01bb8 <attach_command(char const*, int)>, arg=0x7fffffffe28d "1472010", from_tty=1, do_bp_actions=false) at /home/simark/src/binutils-gdb/gdb/main.c:452
+    #22 0x000055555d0f03ad in captured_main_1 (context=0x7fffffffdd10) at /home/simark/src/binutils-gdb/gdb/main.c:1149
+    #23 0x000055555d0f1239 in captured_main (data=0x7fffffffdd10) at /home/simark/src/binutils-gdb/gdb/main.c:1232
+    #24 0x000055555d0f1315 in gdb_main (args=0x7fffffffdd10) at /home/simark/src/binutils-gdb/gdb/main.c:1257
+    #25 0x000055555bb70cf9 in main (argc=7, argv=0x7fffffffde88) at /home/simark/src/binutils-gdb/gdb/gdb.c:32
+
+Catching the exception at this point means that the thread_db_info
+object for this inferior will be left in place, despite the failure to
+load libthread_db.  This means that there won't be further attempts at
+loading libthread_db, because thread_db_load will think that
+libthread_db is already loaded for this inferior and will always exit
+early.  To fix this, add a try/catch around calling try_thread_db_load_1
+in try_thread_db_load, such that if some exception is thrown while
+trying to load libthread_db, we reset / delete the thread_db_info for
+that inferior.  That alone makes attach work fine again, because
+check_for_thread_db is called again in the thread_db_inferior_created
+observer (that happens after we learned about all shared libraries and
+their symbols), and libthread_db is successfully loaded then.
+
+When attaching, I think that the inferior_created observer is a good
+place to try to load libthread_db: it is called once everything has
+stabilized, when we learned about all shared libraries.
+
+The only problem then is that when we first try (and fail) to load
+libthread_db, in reaction to learning about libpthread, we show this
+warning:
+
+    warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
+
+This is misleading, because we do succeed in loading it later.  So when
+attaching, I think we shouldn't try to load libthread_db in reaction to
+the new_objfile events, we should wait until we have learned about all
+shared libraries (using the inferior_created observable).  To do so, add
+an `in_initial_library_scan` flag to struct inferior.  This flag is used
+to postpone loading libthread_db if we are attaching or handling a fork
+child.
+
+When debugging remotely with GDBserver, the same problem happens, except
+that the qSymbol mechanism (allowing the remote side to ask GDB for
+symbols values) is involved.  The fix there is the same idea, we make
+GDB wait until all shared libraries and their symbols are known before
+sending out a qSymbol packet.  This way, we never present the remote
+side a state where libpthread.so's symbols are known but ld-linux's
+symbols aren't.
+
+gdb/ChangeLog:
+
+	* inferior.h (class inferior) <in_initial_library_scan>: New.
+	* infcmd.c (post_create_inferior): Set in_initial_library_scan.
+	* infrun.c (follow_fork_inferior): Likewise.
+	* linux-thread-db.c (try_thread_db_load): Catch exception thrown
+	by try_thread_db_load_1
+	(thread_db_load): Return early if in_initial_library_scan is
+	set.
+	* remote.c (remote_new_objfile): Return early if
+	in_initial_library_scan is set.
+
+Change-Id: I7a279836cfbb2b362b4fde11b196b4aab82f5efb
+
+diff --git a/gdb/infcmd.c b/gdb/infcmd.c
+--- a/gdb/infcmd.c
++++ b/gdb/infcmd.c
+@@ -313,6 +313,10 @@ post_create_inferior (struct target_ops *target, int from_tty)
+       const unsigned solib_add_generation
+ 	= current_program_space->solib_add_generation;
+ 
++      scoped_restore restore_in_initial_library_scan
++	= make_scoped_restore (&current_inferior ()->in_initial_library_scan,
++			       true);
++
+       /* Create the hooks to handle shared library load and unload
+ 	 events.  */
+       solib_create_inferior_hook (from_tty);
+diff --git a/gdb/inferior.h b/gdb/inferior.h
+--- a/gdb/inferior.h
++++ b/gdb/inferior.h
+@@ -511,6 +511,10 @@ class inferior : public refcounted_object
+      architecture/description.  */
+   bool needs_setup = false;
+ 
++  /* True when we are reading the library list of the inferior during an
++     attach or handling a fork child.  */
++  bool in_initial_library_scan = false;
++
+   /* Private data used by the target vector implementation.  */
+   std::unique_ptr<private_inferior> priv;
+ 
+diff --git a/gdb/infrun.c b/gdb/infrun.c
+--- a/gdb/infrun.c
++++ b/gdb/infrun.c
+@@ -540,6 +540,9 @@ holding the child stopped.  Try \"set detach-on-fork\" or \
+ 		 breakpoint.  If a "cloned-VM" event was propagated
+ 		 better throughout the core, this wouldn't be
+ 		 required.  */
++	      scoped_restore restore_in_initial_library_scan
++		= make_scoped_restore (&child_inf->in_initial_library_scan,
++				       true);
+ 	      solib_create_inferior_hook (0);
+ 	    }
+ 	}
+@@ -675,6 +678,8 @@ holding the child stopped.  Try \"set detach-on-fork\" or \
+ 	     shared libraries, and install the solib event breakpoint.
+ 	     If a "cloned-VM" event was propagated better throughout
+ 	     the core, this wouldn't be required.  */
++	  scoped_restore restore_in_initial_library_scan
++	    = make_scoped_restore (&child_inf->in_initial_library_scan, true);
+ 	  solib_create_inferior_hook (0);
+ 	}
+ 
+diff --git a/gdb/linux-thread-db.c b/gdb/linux-thread-db.c
+--- a/gdb/linux-thread-db.c
++++ b/gdb/linux-thread-db.c
+@@ -1012,8 +1012,17 @@ try_thread_db_load (const char *library, bool check_auto_load_safe)
+   if (strchr (library, '/') != NULL)
+     info->filename = gdb_realpath (library).release ();
+ 
+-  if (try_thread_db_load_1 (info))
+-    return true;
++  try
++    {
++      if (try_thread_db_load_1 (info))
++	return true;
++    }
++  catch (const gdb_exception_error &except)
++    {
++      if (libthread_db_debug)
++	exception_fprintf (gdb_stdlog, except,
++			   "Warning: While trying to load libthread_db: ");
++    }
+ 
+   /* This library "refused" to work on current inferior.  */
+   delete_thread_db_info (current_inferior ()->process_target (),
+@@ -1184,10 +1193,15 @@ has_libpthread (void)
+ static bool
+ thread_db_load (void)
+ {
+-  struct thread_db_info *info;
++  inferior *inf = current_inferior ();
+ 
+-  info = get_thread_db_info (current_inferior ()->process_target (),
+-			     inferior_ptid.pid ());
++  /* When attaching / handling fork child, don't try loading libthread_db
++     until we know about all shared libraries.  */
++  if (inf->in_initial_library_scan)
++    return false;
++
++  thread_db_info *info = get_thread_db_info (inf->process_target (),
++					     inferior_ptid.pid ());
+ 
+   if (info != NULL)
+     return true;
+diff --git a/gdb/remote.c b/gdb/remote.c
+--- a/gdb/remote.c
++++ b/gdb/remote.c
+@@ -14299,8 +14299,26 @@ remote_new_objfile (struct objfile *objfile)
+ {
+   remote_target *remote = get_current_remote_target ();
+ 
+-  if (remote != NULL)			/* Have a remote connection.  */
+-    remote->remote_check_symbols ();
++  /* First, check whether the current inferior's process target is a remote
++     target.  */
++  if (remote == nullptr)
++    return;
++
++  /* When we are attaching or handling a fork child and the shared library
++     subsystem reads the list of loaded libraries, we receive new objfile
++     events in between each found library.  The libraries are read in an
++     undefined order, so if we gave the remote side a chance to look up
++     symbols between each objfile, we might give it an inconsistent picture
++     of the inferior.  It could appear that a library A appears loaded but
++     a library B does not, even though library A requires library B.  That
++     would present a state that couldn't normally exist in the inferior.
++
++     So, skip these events, we'll give the remote a chance to look up symbols
++     once all the loaded libraries and their symbols are known to GDB.  */
++    if (current_inferior ()->in_initial_library_scan)
++      return;
++
++  remote->remote_check_symbols ();
+ }
+ 
+ /* Pull all the tracepoints defined on the target and create local

diff --git a/gdb-rhbz1971096-glibc2.34-2.patch b/gdb-rhbz1971096-glibc2.34-2.patch
new file mode 100644
index 0000000..b9edab9
--- /dev/null
+++ b/gdb-rhbz1971096-glibc2.34-2.patch
@@ -0,0 +1,149 @@
+From FEDORA_PATCHES Mon Sep 17 00:00:00 2001
+From: Kevin Buettner <kevinb@redhat.com>
+Date: Mon, 14 Jun 2021 19:20:07 -0700
+Subject: gdb-rhbz1971096-glibc2.34-2.patch
+
+;; Backport patch fixing gdb on glibc-2.34 machines w/ regard to
+;; libthread_db initialization.
+
+libthread_db initialization changes related to upcoming glibc-2.34
+
+This commit makes some adjustments to accomodate the upcoming
+glibc-2.34 release.  Beginning with glibc-2.34, functionality formerly
+contained in libpthread has been moved to libc.  For the time being,
+libpthread.so still exists in the file system, but it won't show up in
+ldd output and therefore won't be able to trigger initialization of
+libthread_db related code.  E.g...
+
+Fedora 34 / glibc-2.33.9000:
+
+[kev@f34-2 gdb]$ ldd testsuite/outputs/gdb.threads/tls/tls
+	linux-vdso.so.1 (0x00007ffcf94fa000)
+	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ff0ba9af000)
+	libm.so.6 => /lib64/libm.so.6 (0x00007ff0ba8d4000)
+	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff0ba8b9000)
+	libc.so.6 => /lib64/libc.so.6 (0x00007ff0ba6c6000)
+	/lib64/ld-linux-x86-64.so.2 (0x00007ff0babf0000)
+
+Fedora 34 / glibc-2.33:
+
+[kev@f34-1 gdb]$ ldd testsuite/outputs/gdb.threads/tls/tls
+	linux-vdso.so.1 (0x00007fff32dc0000)
+	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f815f6de000)
+	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f815f4bf000)
+	libm.so.6 => /lib64/libm.so.6 (0x00007f815f37b000)
+	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f815f360000)
+	libc.so.6 => /lib64/libc.so.6 (0x00007f815f191000)
+	/lib64/ld-linux-x86-64.so.2 (0x00007f815f721000)
+
+Note that libpthread is missing from the ldd output for the
+glibc-2.33.9000 machine.
+
+This means that (unless we happen to think of some entirely different
+mechanism), we'll now need to potentially match "libc" in addition to
+"libpthread" as libraries which might be thread libraries.  This
+accounts for the change made in solib.c.  Note that the new code
+attempts to match "/libc." via strstr().  That trailing dot (".")
+avoids inadvertently matching libraries such as libcrypt (and
+all the other many libraries which begin with "libc").
+
+To avoid attempts to load libthread_db when encountering older
+versions of libc, we now attempt to find "pthread_create" (which is a
+symbol that we'd expect to be in any pthread library) in the
+associated objfile.  This accounts for the changes in
+linux-thread-db.c.
+
+I think that other small adjustments will need to be made elsewhere
+too.  I've been working through regressions on my glibc-2.33.9000
+machine; I've fixed some fairly "obvious" changes in the testsuite
+(which are in other commits).  For the rest, it's not yet clear to me
+whether the handful of remaining failures represent a problem in glibc
+or gdb.  I'm still investigating, however, I'll note that these are
+problems that I only see on my glibc-2.33.9000 machine.
+
+gdb/ChangeLog:
+
+	* solib.c (libpthread_name_p): Match "libc" in addition
+	to "libpthread".
+	* linux-thread-db.c (libpthread_objfile_p): New function.
+	(libpthread_name_p): Adjust preexisting callers to use
+	libpthread_objfile_p().
+
+diff --git a/gdb/linux-thread-db.c b/gdb/linux-thread-db.c
+--- a/gdb/linux-thread-db.c
++++ b/gdb/linux-thread-db.c
+@@ -800,6 +800,24 @@ check_thread_db (struct thread_db_info *info, bool log_progress)
+   return test_passed;
+ }
+ 
++/* Predicate which tests whether objfile OBJ refers to the library
++   containing pthread related symbols.  Historically, this library has
++   been named in such a way that looking for "libpthread" in the name
++   was sufficient to identify it.  As of glibc-2.34, the C library
++   (libc) contains the thread library symbols.  Therefore we check
++   that the name matches a possible thread library, but we also check
++   that it contains at least one of the symbols (pthread_create) that
++   we'd expect to find in the thread library.  */
++
++static bool
++libpthread_objfile_p (objfile *obj)
++{
++  return (libpthread_name_p (objfile_name (obj))
++          && lookup_minimal_symbol ("pthread_create",
++	                            NULL,
++				    obj).minsym != NULL);
++}
++
+ /* Attempt to initialize dlopen()ed libthread_db, described by INFO.
+    Return true on success.
+    Failure could happen if libthread_db does not have symbols we expect,
+@@ -1072,7 +1090,7 @@ try_thread_db_load_from_pdir (const char *subdir)
+     return false;
+ 
+   for (objfile *obj : current_program_space->objfiles ())
+-    if (libpthread_name_p (objfile_name (obj)))
++    if (libpthread_objfile_p (obj))
+       {
+ 	if (try_thread_db_load_from_pdir_1 (obj, subdir))
+ 	  return true;
+@@ -1181,7 +1199,7 @@ static bool
+ has_libpthread (void)
+ {
+   for (objfile *obj : current_program_space->objfiles ())
+-    if (libpthread_name_p (objfile_name (obj)))
++    if (libpthread_objfile_p (obj))
+       return true;
+ 
+   return false;
+@@ -1295,7 +1313,7 @@ thread_db_new_objfile (struct objfile *objfile)
+ 	 of the list of shared libraries to load, and in an app of several
+ 	 thousand shared libraries, this can otherwise be painful.  */
+       && ((objfile->flags & OBJF_MAINLINE) != 0
+-	  || libpthread_name_p (objfile_name (objfile))))
++	  || libpthread_objfile_p (objfile)))
+     check_for_thread_db ();
+ }
+ 
+diff --git a/gdb/solib.c b/gdb/solib.c
+--- a/gdb/solib.c
++++ b/gdb/solib.c
+@@ -906,12 +906,17 @@ Do you need \"set solib-search-path\" or \"set sysroot\"?"),
+ 
+    Uses a fairly simplistic heuristic approach where we check
+    the file name against "/libpthread".  This can lead to false
+-   positives, but this should be good enough in practice.  */
++   positives, but this should be good enough in practice.
++
++   As of glibc-2.34, functions formerly residing in libpthread have
++   been moved to libc, so "/libc." needs to be checked too.  (Matching
++   the "." will avoid matching libraries such as libcrypt.) */
+ 
+ bool
+ libpthread_name_p (const char *name)
+ {
+-  return (strstr (name, "/libpthread") != NULL);
++  return (strstr (name, "/libpthread") != NULL
++          || strstr (name, "/libc.") != NULL );
+ }
+ 
+ /* Return non-zero if SO is the libpthread shared library.  */

diff --git a/gdb-rhbz1971096-glibc2.34-3.patch b/gdb-rhbz1971096-glibc2.34-3.patch
new file mode 100644
index 0000000..5c56504
--- /dev/null
+++ b/gdb-rhbz1971096-glibc2.34-3.patch
@@ -0,0 +1,72 @@
+From FEDORA_PATCHES Mon Sep 17 00:00:00 2001
+From: Kevin Buettner <kevinb@redhat.com>
+Date: Mon, 14 Jun 2021 19:24:41 -0700
+Subject: gdb-rhbz1971096-glibc2.34-3.patch
+
+;; Backport testsuite patch for matching new libthread_db related output
+;; when testing gdb on glibc-2.34 matchines (RH BZ 1971096).
+
+testsuite/glib-2.34: Match/consume optional libthread_db related output
+
+When using glibc-2.34, we now see messages related to the loading of
+the thread library for non-thread programs.  E.g.  for the test case,
+gdb.base/execl-update-breakpoints.exp, we will see the following when
+starting the program:
+
+(gdb) break -qualified main
+Breakpoint 1 at 0x100118c: file /ironwood1/sourceware-git/f34-2-glibc244_fix/bld/../../worktree-glibc244_fix/gdb/testsuite/gdb.base/execl-update-breakpoints.c, line 34.
+(gdb) run
+Starting program: [...]/execl-update-breakpoints1
+[Thread debugging using libthread_db enabled]
+Using host libthread_db library "/lib64/libthread_db.so.1".
+
+The two lines of output related to libthread_db are new; we didn't see
+these in the past.  This is a side effect of libc now containing the
+pthread API - we can no longer tell whether the program is
+multi-threaded by simply looking for libpthread.so.  That said, I
+think that we now want to load libthread_db anyway since it's used to
+resolve TLS variables; i.e. we need it for correctly determining the
+value of errno.
+
+This commit adds the necessary regular expressions to match this
+(optional) additional output in the two tests which were failing
+without it.
+
+gdb/testsuite/ChangeLog:
+
+	* gdb.base/execl-update-breakpoints.exp: Add regular
+	expression for optionally matching output related to
+	libthread_db.
+	* gdb.base/fork-print-inferior-events.exp: Likewise.
+
+diff --git a/gdb/testsuite/gdb.base/execl-update-breakpoints.exp b/gdb/testsuite/gdb.base/execl-update-breakpoints.exp
+--- a/gdb/testsuite/gdb.base/execl-update-breakpoints.exp
++++ b/gdb/testsuite/gdb.base/execl-update-breakpoints.exp
+@@ -132,6 +132,7 @@ proc test { always_inserted } {
+ 	"Continuing\\.\r\n" \
+ 	"${not_nl} is executing new program: ${not_nl}\r\n" \
+ 	"(Reading ${not_nl} from remote target\\.\\.\\.\r\n)*" \
++	"(?:.Thread debugging using .*? enabled.\r\nUsing .*? library .*?\\.\r\n)?" \
+ 	"\r\n" \
+ 	"Breakpoint 1, main.*$gdb_prompt $"
+     set message "continue across exec"
+diff --git a/gdb/testsuite/gdb.base/fork-print-inferior-events.exp b/gdb/testsuite/gdb.base/fork-print-inferior-events.exp
+--- a/gdb/testsuite/gdb.base/fork-print-inferior-events.exp
++++ b/gdb/testsuite/gdb.base/fork-print-inferior-events.exp
+@@ -59,6 +59,7 @@ set detach_child_re "${reading_re}\\\[Detaching after fork from child .*\\\]\r\n
+ set detach_parent_re "${reading_re}\\\[Detaching after fork from parent .*\\\]\r\n"
+ set new_inf_re "${reading_re}\\\[New inferior $decimal \\(.*\\)\\\]\r\n"
+ set inf_detached_re "${reading_re}\\\[Inferior $decimal \\(.*\\) detached\\\]\r\n"
++set thread_db_re "(?:\\\[Thread debugging using .*? enabled\\\]\r\nUsing .*? library .*?\\.\r\n)?"
+ 
+ set expected_output [list \
+ 			 "${attach_child_re}${new_inf_re}${detach_parent_re}${inf_detached_re}" \
+@@ -84,7 +85,7 @@ foreach_with_prefix print_inferior_events { "on" "off" } {
+ 	    set output [lindex $expected_output $i]
+ 	    # Always add the "Starting program..." string so that we
+ 	    # match exactly the lines we want.
+-	    set output "Starting program: $binfile\\s*\r\n${output}${exited_normally_re}"
++	    set output "Starting program: $binfile\\s*\r\n${thread_db_re}${output}${thread_db_re}${exited_normally_re}"
+ 	    set i [expr $i + 1]
+ 	    gdb_test "run" $output
+ 	}

diff --git a/gdb-rhbz1971096-glibc2.34-4.patch b/gdb-rhbz1971096-glibc2.34-4.patch
new file mode 100644
index 0000000..7ce9be5
--- /dev/null
+++ b/gdb-rhbz1971096-glibc2.34-4.patch
@@ -0,0 +1,90 @@
+From FEDORA_PATCHES Mon Sep 17 00:00:00 2001
+From: Kevin Buettner <kevinb@redhat.com>
+Date: Mon, 14 Jun 2021 19:29:19 -0700
+Subject: gdb-rhbz1971096-glibc2.34-4.patch
+
+;; Backport patch adjusting test gdb.base/print-symbol-loading.exp.
+;; (RH BZ 1971096).
+
+print-symbol-loading.exp: Allow libc symbols to be already loaded
+
+One consequence of changing libpthread_name_p() in solib.c to (also)
+match libc is that the symbols for libc will now be loaded by
+solib_add() in solib.c.  I think this is mostly harmless because
+we'll likely want these symbols to be loaded anyway, but it did cause
+two failures in gdb.base/print-symbol-loading.exp.
+
+Specifically...
+
+1)
+
+sharedlibrary .*
+(gdb) PASS: gdb.base/print-symbol-loading.exp: shlib off: load shared-lib
+
+now looks like this:
+
+sharedlibrary .*
+Symbols already loaded for /lib64/libc.so.6
+(gdb) PASS: gdb.base/print-symbol-loading.exp: shlib off: load shared-lib
+
+2)
+
+sharedlibrary .*
+Loading symbols for shared libraries: .*
+(gdb) PASS: gdb.base/print-symbol-loading.exp: shlib brief: load shared-lib
+
+now looks like this:
+
+sharedlibrary .*
+Loading symbols for shared libraries: .*
+Symbols already loaded for /lib64/libc.so.6
+(gdb) PASS: gdb.base/print-symbol-loading.exp: shlib brief: load shared-lib
+
+Fixing case #2 ended up being easier than #1.  #1 had been using
+gdb_test_no_output to correctly match this no-output case.  I
+ended up replacing it with gdb_test_multiple, matching the exact
+expected output for each of the two now acceptable cases.
+
+For case #2, I simply added an optional non-capturing group
+for the potential new output.
+
+gdb/testsuite/ChangeLog:
+
+	* gdb.base/print-symbol-loading.exp (proc test_load_shlib):
+	Allow "Symbols already loaded for..." messages.
+
+diff --git a/gdb/testsuite/gdb.base/print-symbol-loading.exp b/gdb/testsuite/gdb.base/print-symbol-loading.exp
+--- a/gdb/testsuite/gdb.base/print-symbol-loading.exp
++++ b/gdb/testsuite/gdb.base/print-symbol-loading.exp
+@@ -96,6 +96,7 @@ test_load_core full
+ 
+ proc test_load_shlib { print_symbol_loading } {
+     global binfile
++    global gdb_prompt
+     with_test_prefix "shlib ${print_symbol_loading}" {
+ 	clean_restart ${binfile}
+ 	gdb_test_no_output "set auto-solib-add off"
+@@ -106,12 +107,20 @@ proc test_load_shlib { print_symbol_loading } {
+ 	set test_name "load shared-lib"
+ 	switch ${print_symbol_loading} {
+ 	    "off" {
+-		gdb_test_no_output "sharedlibrary .*" \
+-		    ${test_name}
++		set cmd "sharedlibrary .*"
++		set cmd_regex [string_to_regexp $cmd]
++		gdb_test_multiple $cmd $test_name {
++		    -re "^$cmd_regex\r\n$gdb_prompt $" {
++			pass $test_name
++		    }
++		    -re "^$cmd_regex\r\nSymbols already loaded for\[^\r\n\]*\\/libc\\.\[^\r\n\]*\r\n$gdb_prompt $" {
++			pass $test_name
++		    }
++		}
+ 	    }
+ 	    "brief" {
+ 		gdb_test "sharedlibrary .*" \
+-		    "Loading symbols for shared libraries: \\.\\*" \
++		    "Loading symbols for shared libraries: \\.\\*.*?(?:Symbols already loaded for .*?libc)?" \
+ 		    ${test_name}
+ 	    }
+ 	    "full" {

diff --git a/gdb-rhbz1971096-glibc2.34-5.patch b/gdb-rhbz1971096-glibc2.34-5.patch
new file mode 100644
index 0000000..911d2bc
--- /dev/null
+++ b/gdb-rhbz1971096-glibc2.34-5.patch
@@ -0,0 +1,86 @@
+From FEDORA_PATCHES Mon Sep 17 00:00:00 2001
+From: Kevin Buettner <kevinb@redhat.com>
+Date: Mon, 14 Jun 2021 19:32:12 -0700
+Subject: gdb-rhbz1971096-glibc2.34-5.patch
+
+;; Backport patch adjusting test gdb.mi/mi-sym-info.exp (RH BZ 1971096).
+
+mi-sym-info.exp: Increase timeout for 114-symbol-info-functions
+
+Loading libc.so's symbols increased the amount of time needed for
+114-symbol-info-function to fetch symbols, causing a timeout during my
+testing.  I enclosed the entire block with a "with_timeout_factor 4",
+which fixes the problem for me.  (Using 2 also fixed it for me, but it
+might not be enough when running this test on slower machines.)
+
+gdb/testsuite/ChangeLog:
+
+	* gdb.mi/mi-sym-info.exp (114-symbol-info-function test): Increase
+	timeout.
+
+diff --git a/gdb/testsuite/gdb.mi/mi-sym-info.exp b/gdb/testsuite/gdb.mi/mi-sym-info.exp
+--- a/gdb/testsuite/gdb.mi/mi-sym-info.exp
++++ b/gdb/testsuite/gdb.mi/mi-sym-info.exp
+@@ -122,33 +122,35 @@ gdb_test_multiple $cmd $testname -prompt "${mi_gdb_prompt}$" {
+ # (from the symbol table).  There's often so much output output from
+ # this command that we overflow expect's buffers, avoid this by
+ # fetching the output piece by piece.
+-set testname "List all functions"
+-set cmd "114-symbol-info-functions --include-nondebug"
+-set state 0
+-gdb_test_multiple $cmd ${testname} -prompt "${mi_gdb_prompt}$" {
+-    -re "114\\^done,symbols=\{" {
+-	if { $state == 0 } { set state 1 }
+-	exp_continue
+-    }
+-    -re "debug=\\\[${symtab_re}" {
+-	if { $state == 1 } { set state 2 }
+-	exp_continue
+-    }
+-    -re ",${symtab_re}" {
+-	exp_continue
+-    }
+-    -re "\\\],nondebug=\\\[" {
+-	if { $state == 2 } { set state 3 }
+-	exp_continue
+-    }
+-    -re "\{address=${qstr},name=${qstr}\}," {
+-	exp_continue
+-    }
+-    -re "\{address=${qstr},name=${qstr}\}\\\]\}\r\n${mi_gdb_prompt}$" {
+-	if { $state == 3 } {
+-	    pass $gdb_test_name
+-	} else {
+-	    fail $gdb_test_name
++with_timeout_factor 4 {
++    set testname "List all functions"
++    set cmd "114-symbol-info-functions --include-nondebug"
++    set state 0
++    gdb_test_multiple $cmd ${testname} -prompt "${mi_gdb_prompt}$" {
++	-re "114\\^done,symbols=\{" {
++	    if { $state == 0 } { set state 1 }
++	    exp_continue
++	}
++	-re "debug=\\\[${symtab_re}" {
++	    if { $state == 1 } { set state 2 }
++	    exp_continue
++	}
++	-re ",${symtab_re}" {
++	    exp_continue
++	}
++	-re "\\\],nondebug=\\\[" {
++	    if { $state == 2 } { set state 3 }
++	    exp_continue
++	}
++	-re "\{address=${qstr},name=${qstr}\}," {
++	    exp_continue
++	}
++	-re "\{address=${qstr},name=${qstr}\}\\\]\}\r\n${mi_gdb_prompt}$" {
++	    if { $state == 3 } {
++		pass $gdb_test_name
++	    } else {
++		fail $gdb_test_name
++	    }
+ 	}
+     }
+ }

diff --git a/gdb.spec b/gdb.spec
index c369a35..d533f92 100644
--- a/gdb.spec
+++ b/gdb.spec
@@ -37,7 +37,7 @@ Version: 10.2
 
 # The release always contains a leading reserved number, start it at 1.
 # `upstream' is not a part of `name' to stay fully rpm dependencies compatible for the testing.
-Release: 3%{?dist}
+Release: 4%{?dist}
 
 License: GPLv3+ and GPLv3+ with exceptions and GPLv2+ and GPLv2+ with exceptions and GPL+ and LGPLv2+ and LGPLv3+ and BSD and Public Domain and GFDL
 # Do not provide URL for snapshots as the file lasts there only for 2 days.
@@ -1154,6 +1154,10 @@ fi
 %endif
 
 %changelog
+* Mon Jun 14 2021 Kevin Buettner <kevinb@redhat.com> - 10.2-4
+- Backport upstream patches which fix multi-threaded debugging for
+  glibc-2.34 (RHBZ 1971096, Simon Marchi, Kevin Buettner).
+
 * Fri Jun 11 2021 Keith Seitz <keiths@redhat.com> - 10.2-3
 - Backport "Exclude debuginfo files from 'outside ELF segments' warning".
   (Keith Seitz, RH BZ 1898252)

                 reply	other threads:[~2026-06-28  0:00 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=178260481954.1.12383930497775734785.rpms-gdb-71264766c54f@fedoraproject.org \
    --to=kevinb@redhat.com \
    --cc=git-commits@fedoraproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox