Fix RAM usage by splitting deeply nested expressions

[antoyo]

Fix #777

• try only setting TREE_VISITED on the ADDR_EXPR in playback::function::get_address and the ADDR_EXPR in playback::lvalue::get_address. (edit: doesn't work)

I was able to build cargo with 5-10 GB RAM while it used to take +64 GB.

This function would generate deeply nested expressions which made GCC use huge amount of RAM.

[pinskia]

This is due to:

    /* Pre-mark tree nodes with TREE_VISITED so that they can be
       deeply unshared during gimplification (including across
       functions); this requires LANG_HOOKS_DEEP_UNSHARING to be true.  */
    TREE_VISITED (inner) = 1;

In jit-playback.h.

It is NOT GCC that does not like deeply nested expressions BUT rather libgccjit that does not like it.

[antoyo]

I tried this patch and it doesn't help with memory usage:

From 6e5317dacd4930be53d0ea37de0104d009189319 Mon Sep 17 00:00:00 2001
From: Antoni Boucher <[email protected]>
Date: Sun, 26 Oct 2025 10:06:53 -0400
Subject: [PATCH] Fix TREE_VISITED

---
 gcc/jit/jit-playback.cc | 2 ++
 gcc/jit/jit-playback.h  | 5 ++++-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/gcc/jit/jit-playback.cc b/gcc/jit/jit-playback.cc
index e41408b635c..554057255bc 100644
--- a/gcc/jit/jit-playback.cc
+++ b/gcc/jit/jit-playback.cc
@@ -2186,6 +2186,7 @@ get_address (location *loc)
   tree t_thistype = TREE_TYPE (t_lvalue);
   tree t_ptrtype = build_pointer_type (t_thistype);
   tree ptr = fold_build1 (ADDR_EXPR, t_ptrtype, t_lvalue);
+  TREE_VISITED (ptr) = 1;
   if (loc)
     get_context ()->set_tree_location (ptr, loc);
   if (mark_addressable (loc))
@@ -2342,6 +2343,7 @@ playback::function::get_address (location *loc)
   tree t_fndecl = as_fndecl ();
   tree t_fntype = TREE_TYPE (t_fndecl);
   tree t_fnptr = build1 (ADDR_EXPR, build_pointer_type (t_fntype), t_fndecl);
+  TREE_VISITED (t_fnptr) = 1;
   if (loc)
     m_ctxt->set_tree_location (t_fnptr, loc);
   return new rvalue (m_ctxt, t_fnptr);
diff --git a/gcc/jit/jit-playback.h b/gcc/jit/jit-playback.h
index 6f297136edc..fb1914e8bc7 100644
--- a/gcc/jit/jit-playback.h
+++ b/gcc/jit/jit-playback.h
@@ -757,7 +757,10 @@ public:
     /* Pre-mark tree nodes with TREE_VISITED so that they can be
        deeply unshared during gimplification (including across
        functions); this requires LANG_HOOKS_DEEP_UNSHARING to be true.  */
-    TREE_VISITED (inner) = 1;
+    // TODO: try only setting TREE_VISITED on the ADDR_EXPR in
+    // playback::function::get_address and the ADDR_EXPR in
+    // playback::lvalue::get_address.
+    //TREE_VISITED (inner) = 1;
   }
 
   rvalue *
-- 
2.51.1

[antoyo]

If this helps, the memory usage happens in gimplify_function_tree.