LCOV - code coverage report
Current view: top level - gcc - ipa-split.c (source / functions) Hit Total Coverage
Test: gcc.info Lines: 816 922 88.5 %
Date: 2020-03-28 11:57:23 Functions: 20 21 95.2 %
Legend: Lines: hit not hit | Branches: + taken - not taken # not executed Branches: 0 0 -

           Branch data     Line data    Source code
       1                 :            : /* Function splitting pass
       2                 :            :    Copyright (C) 2010-2020 Free Software Foundation, Inc.
       3                 :            :    Contributed by Jan Hubicka  <jh@suse.cz>
       4                 :            : 
       5                 :            : This file is part of GCC.
       6                 :            : 
       7                 :            : GCC is free software; you can redistribute it and/or modify it under
       8                 :            : the terms of the GNU General Public License as published by the Free
       9                 :            : Software Foundation; either version 3, or (at your option) any later
      10                 :            : version.
      11                 :            : 
      12                 :            : GCC is distributed in the hope that it will be useful, but WITHOUT ANY
      13                 :            : WARRANTY; without even the implied warranty of MERCHANTABILITY or
      14                 :            : FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
      15                 :            : for more details.
      16                 :            : 
      17                 :            : You should have received a copy of the GNU General Public License
      18                 :            : along with GCC; see the file COPYING3.  If not see
      19                 :            : <http://www.gnu.org/licenses/>.  */
      20                 :            : 
      21                 :            : /* The purpose of this pass is to split function bodies to improve
      22                 :            :    inlining.  I.e. for function of the form:
      23                 :            : 
      24                 :            :    func (...)
      25                 :            :      {
      26                 :            :        if (cheap_test)
      27                 :            :          something_small
      28                 :            :        else
      29                 :            :          something_big
      30                 :            :      }
      31                 :            : 
      32                 :            :    Produce:
      33                 :            : 
      34                 :            :    func.part (...)
      35                 :            :      {
      36                 :            :         something_big
      37                 :            :      }
      38                 :            : 
      39                 :            :    func (...)
      40                 :            :      {
      41                 :            :        if (cheap_test)
      42                 :            :          something_small
      43                 :            :        else
      44                 :            :          func.part (...);
      45                 :            :      }
      46                 :            : 
      47                 :            :    When func becomes inlinable and when cheap_test is often true, inlining func,
      48                 :            :    but not fund.part leads to performance improvement similar as inlining
      49                 :            :    original func while the code size growth is smaller.
      50                 :            : 
      51                 :            :    The pass is organized in three stages:
      52                 :            :    1) Collect local info about basic block into BB_INFO structure and
      53                 :            :       compute function body estimated size and time.
      54                 :            :    2) Via DFS walk find all possible basic blocks where we can split
      55                 :            :       and chose best one.
      56                 :            :    3) If split point is found, split at the specified BB by creating a clone
      57                 :            :       and updating function to call it.  
      58                 :            : 
      59                 :            :    The decisions what functions to split are in execute_split_functions
      60                 :            :    and consider_split.  
      61                 :            : 
      62                 :            :    There are several possible future improvements for this pass including:
      63                 :            : 
      64                 :            :    1) Splitting to break up large functions
      65                 :            :    2) Splitting to reduce stack frame usage
      66                 :            :    3) Allow split part of function to use values computed in the header part.
      67                 :            :       The values needs to be passed to split function, perhaps via same
      68                 :            :       interface as for nested functions or as argument.
      69                 :            :    4) Support for simple rematerialization.  I.e. when split part use
      70                 :            :       value computed in header from function parameter in very cheap way, we
      71                 :            :       can just recompute it.
      72                 :            :    5) Support splitting of nested functions.
      73                 :            :    6) Support non-SSA arguments.  
      74                 :            :    7) There is nothing preventing us from producing multiple parts of single function
      75                 :            :       when needed or splitting also the parts.  */
      76                 :            : 
      77                 :            : #include "config.h"
      78                 :            : #include "system.h"
      79                 :            : #include "coretypes.h"
      80                 :            : #include "backend.h"
      81                 :            : #include "rtl.h"
      82                 :            : #include "tree.h"
      83                 :            : #include "gimple.h"
      84                 :            : #include "cfghooks.h"
      85                 :            : #include "alloc-pool.h"
      86                 :            : #include "tree-pass.h"
      87                 :            : #include "ssa.h"
      88                 :            : #include "cgraph.h"
      89                 :            : #include "diagnostic.h"
      90                 :            : #include "fold-const.h"
      91                 :            : #include "cfganal.h"
      92                 :            : #include "calls.h"
      93                 :            : #include "gimplify.h"
      94                 :            : #include "gimple-iterator.h"
      95                 :            : #include "gimplify-me.h"
      96                 :            : #include "gimple-walk.h"
      97                 :            : #include "symbol-summary.h"
      98                 :            : #include "ipa-prop.h"
      99                 :            : #include "tree-cfg.h"
     100                 :            : #include "tree-into-ssa.h"
     101                 :            : #include "tree-dfa.h"
     102                 :            : #include "tree-inline.h"
     103                 :            : #include "gimple-pretty-print.h"
     104                 :            : #include "ipa-fnsummary.h"
     105                 :            : #include "cfgloop.h"
     106                 :            : #include "attribs.h"
     107                 :            : 
     108                 :            : /* Per basic block info.  */
     109                 :            : 
     110                 :    5331970 : class split_bb_info
     111                 :            : {
     112                 :            : public:
     113                 :            :   unsigned int size;
     114                 :            :   sreal time;
     115                 :            : };
     116                 :            : 
     117                 :            : static vec<split_bb_info> bb_info_vec;
     118                 :            : 
     119                 :            : /* Description of split point.  */
     120                 :            : 
     121                 :     926867 : class split_point
     122                 :            : {
     123                 :            : public:
     124                 :            :   /* Size of the partitions.  */
     125                 :            :   sreal header_time, split_time;
     126                 :            :   unsigned int header_size, split_size;
     127                 :            : 
     128                 :            :   /* SSA names that need to be passed into spit function.  */
     129                 :            :   bitmap ssa_names_to_pass;
     130                 :            : 
     131                 :            :   /* Basic block where we split (that will become entry point of new function.  */
     132                 :            :   basic_block entry_bb;
     133                 :            : 
     134                 :            :   /* Count for entering the split part.
     135                 :            :      This is not count of the entry_bb because it may be in loop.  */
     136                 :            :   profile_count count;
     137                 :            : 
     138                 :            :   /* Basic blocks we are splitting away.  */
     139                 :            :   bitmap split_bbs;
     140                 :            : 
     141                 :            :   /* True when return value is computed on split part and thus it needs
     142                 :            :      to be returned.  */
     143                 :            :   bool split_part_set_retval;
     144                 :            : };
     145                 :            : 
     146                 :            : /* Best split point found.  */
     147                 :            : 
     148                 :            : class split_point best_split_point;
     149                 :            : 
     150                 :            : /* Set of basic blocks that are not allowed to dominate a split point.  */
     151                 :            : 
     152                 :            : static bitmap forbidden_dominators;
     153                 :            : 
     154                 :            : static tree find_retval (basic_block return_bb);
     155                 :            : 
     156                 :            : /* Callback for walk_stmt_load_store_addr_ops.  If T is non-SSA automatic
     157                 :            :    variable, check it if it is present in bitmap passed via DATA.  */
     158                 :            : 
     159                 :            : static bool
     160                 :      10043 : test_nonssa_use (gimple *, tree t, tree, void *data)
     161                 :            : {
     162                 :      10043 :   t = get_base_address (t);
     163                 :            : 
     164                 :      10043 :   if (!t || is_gimple_reg (t))
     165                 :          0 :     return false;
     166                 :            : 
     167                 :      10043 :   if (TREE_CODE (t) == PARM_DECL
     168                 :       9999 :       || (VAR_P (t)
     169                 :       5099 :           && auto_var_in_fn_p (t, current_function_decl))
     170                 :       8534 :       || TREE_CODE (t) == RESULT_DECL
     171                 :            :          /* Normal labels are part of CFG and will be handled gratefully.
     172                 :            :             Forced labels however can be used directly by statements and
     173                 :            :             need to stay in one partition along with their uses.  */
     174                 :      18575 :       || (TREE_CODE (t) == LABEL_DECL
     175                 :        323 :           && FORCED_LABEL (t)))
     176                 :       1515 :     return bitmap_bit_p ((bitmap)data, DECL_UID (t));
     177                 :            : 
     178                 :            :   /* For DECL_BY_REFERENCE, the return value is actually a pointer.  We want
     179                 :            :      to pretend that the value pointed to is actual result decl.  */
     180                 :       8528 :   if ((TREE_CODE (t) == MEM_REF || INDIRECT_REF_P (t))
     181                 :       3823 :       && TREE_CODE (TREE_OPERAND (t, 0)) == SSA_NAME
     182                 :       3819 :       && SSA_NAME_VAR (TREE_OPERAND (t, 0))
     183                 :       3510 :       && TREE_CODE (SSA_NAME_VAR (TREE_OPERAND (t, 0))) == RESULT_DECL
     184                 :       8528 :       && DECL_BY_REFERENCE (DECL_RESULT (current_function_decl)))
     185                 :          0 :     return
     186                 :          0 :       bitmap_bit_p ((bitmap)data,
     187                 :          0 :                     DECL_UID (DECL_RESULT (current_function_decl)));
     188                 :            : 
     189                 :            :   return false;
     190                 :            : }
     191                 :            : 
     192                 :            : /* Dump split point CURRENT.  */
     193                 :            : 
     194                 :            : static void
     195                 :         11 : dump_split_point (FILE * file, class split_point *current)
     196                 :            : {
     197                 :         11 :   fprintf (file,
     198                 :            :            "Split point at BB %i\n"
     199                 :            :            "  header time: %f header size: %i\n"
     200                 :            :            "  split time: %f split size: %i\n  bbs: ",
     201                 :         11 :            current->entry_bb->index, current->header_time.to_double (),
     202                 :            :            current->header_size, current->split_time.to_double (),
     203                 :            :            current->split_size);
     204                 :         11 :   dump_bitmap (file, current->split_bbs);
     205                 :         11 :   fprintf (file, "  SSA names to pass: ");
     206                 :         11 :   dump_bitmap (file, current->ssa_names_to_pass);
     207                 :         11 : }
     208                 :            : 
     209                 :            : /* Look for all BBs in header that might lead to the split part and verify
     210                 :            :    that they are not defining any non-SSA var used by the split part.
     211                 :            :    Parameters are the same as for consider_split.  */
     212                 :            : 
     213                 :            : static bool
     214                 :       5384 : verify_non_ssa_vars (class split_point *current, bitmap non_ssa_vars,
     215                 :            :                      basic_block return_bb)
     216                 :            : {
     217                 :       5384 :   bitmap seen = BITMAP_ALLOC (NULL);
     218                 :       5384 :   vec<basic_block> worklist = vNULL;
     219                 :       5384 :   edge e;
     220                 :       5384 :   edge_iterator ei;
     221                 :       5384 :   bool ok = true;
     222                 :       5384 :   basic_block bb;
     223                 :            : 
     224                 :      12154 :   FOR_EACH_EDGE (e, ei, current->entry_bb->preds)
     225                 :       6770 :     if (e->src != ENTRY_BLOCK_PTR_FOR_FN (cfun)
     226                 :       6770 :         && !bitmap_bit_p (current->split_bbs, e->src->index))
     227                 :            :       {
     228                 :       6567 :         worklist.safe_push (e->src);
     229                 :       6567 :         bitmap_set_bit (seen, e->src->index);
     230                 :            :       }
     231                 :            : 
     232                 :      13432 :   while (!worklist.is_empty ())
     233                 :            :     {
     234                 :       8949 :       bb = worklist.pop ();
     235                 :      18166 :       FOR_EACH_EDGE (e, ei, bb->preds)
     236                 :       9217 :         if (e->src != ENTRY_BLOCK_PTR_FOR_FN (cfun)
     237                 :       9217 :             && bitmap_set_bit (seen, e->src->index))
     238                 :            :           {
     239                 :       2669 :             gcc_checking_assert (!bitmap_bit_p (current->split_bbs,
     240                 :            :                                                 e->src->index));
     241                 :       2669 :             worklist.safe_push (e->src);
     242                 :            :           }
     243                 :      94493 :       for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi);
     244                 :      76595 :            gsi_next (&bsi))
     245                 :            :         {
     246                 :      77496 :           gimple *stmt = gsi_stmt (bsi);
     247                 :      77496 :           if (is_gimple_debug (stmt))
     248                 :      55413 :             continue;
     249                 :      22083 :           if (walk_stmt_load_store_addr_ops
     250                 :      22083 :               (stmt, non_ssa_vars, test_nonssa_use, test_nonssa_use,
     251                 :            :                test_nonssa_use))
     252                 :            :             {
     253                 :        901 :               ok = false;
     254                 :        901 :               goto done;
     255                 :            :             }
     256                 :      76601 :           if (glabel *label_stmt = dyn_cast <glabel *> (stmt))
     257                 :          6 :             if (test_nonssa_use (stmt, gimple_label_label (label_stmt),
     258                 :            :                                  NULL_TREE, non_ssa_vars))
     259                 :            :               {
     260                 :          0 :                 ok = false;
     261                 :          0 :                 goto done;
     262                 :            :               }
     263                 :            :         }
     264                 :       8358 :       for (gphi_iterator bsi = gsi_start_phis (bb); !gsi_end_p (bsi);
     265                 :        310 :            gsi_next (&bsi))
     266                 :            :         {
     267                 :        310 :           if (walk_stmt_load_store_addr_ops
     268                 :        310 :               (gsi_stmt (bsi), non_ssa_vars, test_nonssa_use, test_nonssa_use,
     269                 :            :                test_nonssa_use))
     270                 :            :             {
     271                 :          0 :               ok = false;
     272                 :          0 :               goto done;
     273                 :            :             }
     274                 :            :         }
     275                 :      22790 :       FOR_EACH_EDGE (e, ei, bb->succs)
     276                 :            :         {
     277                 :      14742 :           if (e->dest != return_bb)
     278                 :      13761 :             continue;
     279                 :        981 :           for (gphi_iterator bsi = gsi_start_phis (return_bb);
     280                 :       2192 :                !gsi_end_p (bsi);
     281                 :       1211 :                gsi_next (&bsi))
     282                 :            :             {
     283                 :       1211 :               gphi *stmt = bsi.phi ();
     284                 :       1211 :               tree op = gimple_phi_arg_def (stmt, e->dest_idx);
     285                 :            : 
     286                 :       2422 :               if (virtual_operand_p (gimple_phi_result (stmt)))
     287                 :        934 :                 continue;
     288                 :        277 :               if (TREE_CODE (op) != SSA_NAME
     289                 :        277 :                   && test_nonssa_use (stmt, op, op, non_ssa_vars))
     290                 :            :                 {
     291                 :          0 :                   ok = false;
     292                 :          0 :                   goto done;
     293                 :            :                 }
     294                 :            :             }
     295                 :            :         }
     296                 :            :     }
     297                 :            : 
     298                 :            :   /* Verify that the rest of function does not define any label
     299                 :            :      used by the split part.  */
     300                 :      69669 :   FOR_EACH_BB_FN (bb, cfun)
     301                 :      65189 :     if (!bitmap_bit_p (current->split_bbs, bb->index)
     302                 :      65189 :         && !bitmap_bit_p (seen, bb->index))
     303                 :            :       {
     304                 :       9804 :         gimple_stmt_iterator bsi;
     305                 :      19922 :         for (bsi = gsi_start_bb (bb); !gsi_end_p (bsi); gsi_next (&bsi))
     306                 :      10118 :           if (glabel *label_stmt = dyn_cast <glabel *> (gsi_stmt (bsi)))
     307                 :            :             {
     308                 :        317 :               if (test_nonssa_use (label_stmt,
     309                 :            :                                    gimple_label_label (label_stmt),
     310                 :            :                                    NULL_TREE, non_ssa_vars))
     311                 :            :                 {
     312                 :          3 :                   ok = false;
     313                 :          3 :                   goto done;
     314                 :            :                 }
     315                 :            :             }
     316                 :            :           else
     317                 :            :             break;
     318                 :            :       }
     319                 :            :     
     320                 :       4480 : done:
     321                 :       5384 :   BITMAP_FREE (seen);
     322                 :       5384 :   worklist.release ();
     323                 :       5384 :   return ok;
     324                 :            : }
     325                 :            : 
     326                 :            : /* If STMT is a call, check the callee against a list of forbidden
     327                 :            :    predicate functions.  If a match is found, look for uses of the
     328                 :            :    call result in condition statements that compare against zero.
     329                 :            :    For each such use, find the block targeted by the condition
     330                 :            :    statement for the nonzero result, and set the bit for this block
     331                 :            :    in the forbidden dominators bitmap.  The purpose of this is to avoid
     332                 :            :    selecting a split point where we are likely to lose the chance
     333                 :            :    to optimize away an unused function call.  */
     334                 :            : 
     335                 :            : static void
     336                 :   26233300 : check_forbidden_calls (gimple *stmt)
     337                 :            : {
     338                 :   26233300 :   imm_use_iterator use_iter;
     339                 :   26233300 :   use_operand_p use_p;
     340                 :   26233300 :   tree lhs;
     341                 :            : 
     342                 :            :   /* At the moment, __builtin_constant_p is the only forbidden
     343                 :            :      predicate function call (see PR49642).  */
     344                 :   26233300 :   if (!gimple_call_builtin_p (stmt, BUILT_IN_CONSTANT_P))
     345                 :   26232700 :     return;
     346                 :            : 
     347                 :        652 :   lhs = gimple_call_lhs (stmt);
     348                 :            : 
     349                 :        652 :   if (!lhs || TREE_CODE (lhs) != SSA_NAME)
     350                 :            :     return;
     351                 :            : 
     352                 :       1798 :   FOR_EACH_IMM_USE_FAST (use_p, use_iter, lhs)
     353                 :            :     {
     354                 :       1146 :       tree op1;
     355                 :       1146 :       basic_block use_bb, forbidden_bb;
     356                 :       1146 :       enum tree_code code;
     357                 :       1146 :       edge true_edge, false_edge;
     358                 :       1146 :       gcond *use_stmt;
     359                 :            : 
     360                 :       1146 :       use_stmt = dyn_cast <gcond *> (USE_STMT (use_p));
     361                 :       1146 :       if (!use_stmt)
     362                 :         26 :         continue;
     363                 :            : 
     364                 :            :       /* Assuming canonical form for GIMPLE_COND here, with constant
     365                 :            :          in second position.  */
     366                 :       1120 :       op1 = gimple_cond_rhs (use_stmt);
     367                 :       1120 :       code = gimple_cond_code (use_stmt);
     368                 :       1120 :       use_bb = gimple_bb (use_stmt);
     369                 :            : 
     370                 :       1120 :       extract_true_false_edges_from_block (use_bb, &true_edge, &false_edge);
     371                 :            : 
     372                 :            :       /* We're only interested in comparisons that distinguish
     373                 :            :          unambiguously from zero.  */
     374                 :       1120 :       if (!integer_zerop (op1) || code == LE_EXPR || code == GE_EXPR)
     375                 :          0 :         continue;
     376                 :            : 
     377                 :       1120 :       if (code == EQ_EXPR)
     378                 :          5 :         forbidden_bb = false_edge->dest;
     379                 :            :       else
     380                 :       1115 :         forbidden_bb = true_edge->dest;
     381                 :            : 
     382                 :       1120 :       bitmap_set_bit (forbidden_dominators, forbidden_bb->index);
     383                 :            :     }
     384                 :            : }
     385                 :            : 
     386                 :            : /* If BB is dominated by any block in the forbidden dominators set,
     387                 :            :    return TRUE; else FALSE.  */
     388                 :            : 
     389                 :            : static bool
     390                 :         21 : dominated_by_forbidden (basic_block bb)
     391                 :            : {
     392                 :         21 :   unsigned dom_bb;
     393                 :         21 :   bitmap_iterator bi;
     394                 :            : 
     395                 :         35 :   EXECUTE_IF_SET_IN_BITMAP (forbidden_dominators, 1, dom_bb, bi)
     396                 :            :     {
     397                 :         28 :       if (dominated_by_p (CDI_DOMINATORS, bb,
     398                 :         28 :                           BASIC_BLOCK_FOR_FN (cfun, dom_bb)))
     399                 :            :         return true;
     400                 :            :     }
     401                 :            : 
     402                 :            :   return false;
     403                 :            : }
     404                 :            : 
     405                 :            : /* For give split point CURRENT and return block RETURN_BB return 1
     406                 :            :    if ssa name VAL is set by split part and 0 otherwise.  */
     407                 :            : static bool
     408                 :      14253 : split_part_set_ssa_name_p (tree val, class split_point *current,
     409                 :            :                            basic_block return_bb)
     410                 :            : {
     411                 :      14253 :   if (TREE_CODE (val) != SSA_NAME)
     412                 :            :     return false;
     413                 :            : 
     414                 :      14253 :   return (!SSA_NAME_IS_DEFAULT_DEF (val)
     415                 :      14253 :           && (bitmap_bit_p (current->split_bbs,
     416                 :      11816 :                             gimple_bb (SSA_NAME_DEF_STMT (val))->index)
     417                 :      11816 :               || gimple_bb (SSA_NAME_DEF_STMT (val)) == return_bb));
     418                 :            : }
     419                 :            : 
     420                 :            : /* We found an split_point CURRENT.  NON_SSA_VARS is bitmap of all non ssa
     421                 :            :    variables used and RETURN_BB is return basic block.
     422                 :            :    See if we can split function here.  */
     423                 :            : 
     424                 :            : static void
     425                 :    1281160 : consider_split (class split_point *current, bitmap non_ssa_vars,
     426                 :            :                 basic_block return_bb)
     427                 :            : {
     428                 :    1281160 :   tree parm;
     429                 :    1281160 :   unsigned int num_args = 0;
     430                 :    1281160 :   unsigned int call_overhead;
     431                 :    1281160 :   edge e;
     432                 :    1281160 :   edge_iterator ei;
     433                 :    1281160 :   gphi_iterator bsi;
     434                 :    1281160 :   unsigned int i;
     435                 :    1281160 :   tree retval;
     436                 :    1281160 :   bool back_edge = false;
     437                 :            : 
     438                 :    1281160 :   if (dump_file && (dump_flags & TDF_DETAILS))
     439                 :          4 :     dump_split_point (dump_file, current);
     440                 :            : 
     441                 :    1281160 :   current->count = profile_count::zero ();
     442                 :    2807370 :   FOR_EACH_EDGE (e, ei, current->entry_bb->preds)
     443                 :            :     {
     444                 :    1526210 :       if (e->flags & EDGE_DFS_BACK)
     445                 :      63649 :         back_edge = true;
     446                 :    1526210 :       if (!bitmap_bit_p (current->split_bbs, e->src->index))
     447                 :    1462560 :         current->count += e->count ();
     448                 :            :     }
     449                 :            : 
     450                 :            :   /* Do not split when we would end up calling function anyway.
     451                 :            :      Compares are three state, use !(...<...) to also give up when outcome
     452                 :            :      is unknown.  */
     453                 :    2562310 :   if (!(current->count
     454                 :    1281160 :        < (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count.apply_scale
     455                 :    2562310 :            (param_partial_inlining_entry_probability, 100))))
     456                 :            :     {
     457                 :            :       /* When profile is guessed, we cannot expect it to give us
     458                 :            :          realistic estimate on likeliness of function taking the
     459                 :            :          complex path.  As a special case, when tail of the function is
     460                 :            :          a loop, enable splitting since inlining code skipping the loop
     461                 :            :          is likely noticeable win.  */
     462                 :     881298 :       if (back_edge
     463                 :      52725 :           && profile_status_for_fn (cfun) != PROFILE_READ
     464                 :     933989 :           && current->count
     465                 :      52691 :                  < ENTRY_BLOCK_PTR_FOR_FN (cfun)->count)
     466                 :            :         {
     467                 :       9053 :           if (dump_file && (dump_flags & TDF_DETAILS))
     468                 :            :             {
     469                 :          0 :               fprintf (dump_file,
     470                 :            :                        "  Split before loop, accepting despite low counts");
     471                 :          0 :               current->count.dump (dump_file);
     472                 :          0 :               fprintf (dump_file, " ");
     473                 :          0 :               ENTRY_BLOCK_PTR_FOR_FN (cfun)->count.dump (dump_file);
     474                 :            :             }
     475                 :            :         }
     476                 :            :       else
     477                 :            :         {
     478                 :     872245 :           if (dump_file && (dump_flags & TDF_DETAILS))
     479                 :          2 :             fprintf (dump_file,
     480                 :            :                      "  Refused: incoming frequency is too large.\n");
     481                 :    1248480 :           return;
     482                 :            :         }
     483                 :            :     }
     484                 :            : 
     485                 :     408912 :   if (!current->header_size)
     486                 :            :     {
     487                 :          0 :       if (dump_file && (dump_flags & TDF_DETAILS))
     488                 :          0 :         fprintf (dump_file, "  Refused: header empty\n");
     489                 :          0 :       return;
     490                 :            :     }
     491                 :            : 
     492                 :            :   /* Verify that PHI args on entry are either virtual or all their operands
     493                 :            :      incoming from header are the same.  */
     494                 :     484699 :   for (bsi = gsi_start_phis (current->entry_bb); !gsi_end_p (bsi); gsi_next (&bsi))
     495                 :            :     {
     496                 :      83445 :       gphi *stmt = bsi.phi ();
     497                 :      83445 :       tree val = NULL;
     498                 :            : 
     499                 :     166890 :       if (virtual_operand_p (gimple_phi_result (stmt)))
     500                 :      33207 :         continue;
     501                 :     132195 :       for (i = 0; i < gimple_phi_num_args (stmt); i++)
     502                 :            :         {
     503                 :      89615 :           edge e = gimple_phi_arg_edge (stmt, i);
     504                 :      89615 :           if (!bitmap_bit_p (current->split_bbs, e->src->index))
     505                 :            :             {
     506                 :      59899 :               tree edge_val = gimple_phi_arg_def (stmt, i);
     507                 :      59899 :               if (val && edge_val != val)
     508                 :            :                 {
     509                 :       7658 :                   if (dump_file && (dump_flags & TDF_DETAILS))
     510                 :          0 :                     fprintf (dump_file,
     511                 :            :                              "  Refused: entry BB has PHI with multiple variants\n");
     512                 :       7658 :                   return;
     513                 :            :                 }
     514                 :            :               val = edge_val;
     515                 :            :             }
     516                 :            :         }
     517                 :            :     }
     518                 :            : 
     519                 :            : 
     520                 :            :   /* See what argument we will pass to the split function and compute
     521                 :            :      call overhead.  */
     522                 :     401254 :   call_overhead = eni_size_weights.call_cost;
     523                 :    2244040 :   for (parm = DECL_ARGUMENTS (current_function_decl); parm;
     524                 :     921392 :        parm = DECL_CHAIN (parm))
     525                 :            :     {
     526                 :     921392 :       if (!is_gimple_reg (parm))
     527                 :            :         {
     528                 :      57706 :           if (bitmap_bit_p (non_ssa_vars, DECL_UID (parm)))
     529                 :            :             {
     530                 :          0 :               if (dump_file && (dump_flags & TDF_DETAILS))
     531                 :          0 :                 fprintf (dump_file,
     532                 :            :                          "  Refused: need to pass non-ssa param values\n");
     533                 :          0 :               return;
     534                 :            :             }
     535                 :            :         }
     536                 :            :       else
     537                 :            :         {
     538                 :     863686 :           tree ddef = ssa_default_def (cfun, parm);
     539                 :     863686 :           if (ddef
     540                 :    1702030 :               && bitmap_bit_p (current->ssa_names_to_pass,
     541                 :     838341 :                                SSA_NAME_VERSION (ddef)))
     542                 :            :             {
     543                 :     329124 :               if (!VOID_TYPE_P (TREE_TYPE (parm)))
     544                 :     329124 :                 call_overhead += estimate_move_cost (TREE_TYPE (parm), false);
     545                 :     329124 :               num_args++;
     546                 :            :             }
     547                 :            :         }
     548                 :            :     }
     549                 :     401254 :   if (!VOID_TYPE_P (TREE_TYPE (current_function_decl)))
     550                 :     401254 :     call_overhead += estimate_move_cost (TREE_TYPE (current_function_decl),
     551                 :            :                                          false);
     552                 :            : 
     553                 :     401254 :   if (current->split_size <= call_overhead)
     554                 :            :     {
     555                 :     212799 :       if (dump_file && (dump_flags & TDF_DETAILS))
     556                 :          0 :         fprintf (dump_file,
     557                 :            :                  "  Refused: split size is smaller than call overhead\n");
     558                 :     212799 :       return;
     559                 :            :     }
     560                 :            :   /* FIXME: The logic here is not very precise, because inliner does use
     561                 :            :      inline predicates to reduce function body size.  We add 10 to anticipate
     562                 :            :      that.  Next stage1 we should try to be more meaningful here.  */
     563                 :     188455 :   if (current->header_size + call_overhead
     564                 :     188455 :       >= (unsigned int)(DECL_DECLARED_INLINE_P (current_function_decl)
     565                 :      46131 :                         ? param_max_inline_insns_single
     566                 :     188455 :                         : param_max_inline_insns_auto) + 10)
     567                 :            :     {
     568                 :     115958 :       if (dump_file && (dump_flags & TDF_DETAILS))
     569                 :          0 :         fprintf (dump_file,
     570                 :            :                  "  Refused: header size is too large for inline candidate\n");
     571                 :     115958 :       return;
     572                 :            :     }
     573                 :            : 
     574                 :            :   /* Splitting functions brings the target out of comdat group; this will
     575                 :            :      lead to code duplication if the function is reused by other unit.
     576                 :            :      Limit this duplication.  This is consistent with limit in tree-sra.c  
     577                 :            :      FIXME: with LTO we ought to be able to do better!  */
     578                 :     120299 :   if (DECL_ONE_ONLY (current_function_decl)
     579                 :     120298 :       && current->split_size >= (unsigned int) param_max_inline_insns_auto + 10)
     580                 :            :     {
     581                 :       4293 :       if (dump_file && (dump_flags & TDF_DETAILS))
     582                 :          0 :         fprintf (dump_file,
     583                 :            :                  "  Refused: function is COMDAT and tail is too large\n");
     584                 :       4293 :       return;
     585                 :            :     }
     586                 :            :   /* For comdat functions also reject very small tails; those will likely get
     587                 :            :      inlined back and we do not want to risk the duplication overhead.
     588                 :            :      FIXME: with LTO we ought to be able to do better!  */
     589                 :     111713 :   if (DECL_ONE_ONLY (current_function_decl)
     590                 :      68204 :       && current->split_size
     591                 :      43508 :          <= (unsigned int) param_early_inlining_insns / 2)
     592                 :            :     {
     593                 :       5303 :       if (dump_file && (dump_flags & TDF_DETAILS))
     594                 :          0 :         fprintf (dump_file,
     595                 :            :                  "  Refused: function is COMDAT and tail is too small\n");
     596                 :       5303 :       return;
     597                 :            :     }
     598                 :            : 
     599                 :            :   /* FIXME: we currently can pass only SSA function parameters to the split
     600                 :            :      arguments.  Once parm_adjustment infrastructure is supported by cloning,
     601                 :            :      we can pass more than that.  */
     602                 :      62901 :   if (num_args != bitmap_count_bits (current->ssa_names_to_pass))
     603                 :            :     {
     604                 :            :       
     605                 :      29307 :       if (dump_file && (dump_flags & TDF_DETAILS))
     606                 :          0 :         fprintf (dump_file,
     607                 :            :                  "  Refused: need to pass non-param values\n");
     608                 :      29307 :       return;
     609                 :            :     }
     610                 :            : 
     611                 :            :   /* When there are non-ssa vars used in the split region, see if they
     612                 :            :      are used in the header region.  If so, reject the split.
     613                 :            :      FIXME: we can use nested function support to access both.  */
     614                 :      33594 :   if (!bitmap_empty_p (non_ssa_vars)
     615                 :      33594 :       && !verify_non_ssa_vars (current, non_ssa_vars, return_bb))
     616                 :            :     {
     617                 :        904 :       if (dump_file && (dump_flags & TDF_DETAILS))
     618                 :          0 :         fprintf (dump_file,
     619                 :            :                  "  Refused: split part has non-ssa uses\n");
     620                 :        904 :       return;
     621                 :            :     }
     622                 :            : 
     623                 :            :   /* If the split point is dominated by a forbidden block, reject
     624                 :            :      the split.  */
     625                 :      32690 :   if (!bitmap_empty_p (forbidden_dominators)
     626                 :      32690 :       && dominated_by_forbidden (current->entry_bb))
     627                 :            :     {
     628                 :         14 :       if (dump_file && (dump_flags & TDF_DETAILS))
     629                 :          0 :         fprintf (dump_file,
     630                 :            :                  "  Refused: split point dominated by forbidden block\n");
     631                 :         14 :       return;
     632                 :            :     }
     633                 :            : 
     634                 :            :   /* See if retval used by return bb is computed by header or split part.
     635                 :            :      When it is computed by split part, we need to produce return statement
     636                 :            :      in the split part and add code to header to pass it around.
     637                 :            : 
     638                 :            :      This is bit tricky to test:
     639                 :            :        1) When there is no return_bb or no return value, we always pass
     640                 :            :           value around.
     641                 :            :        2) Invariants are always computed by caller.
     642                 :            :        3) For SSA we need to look if defining statement is in header or split part
     643                 :            :        4) For non-SSA we need to look where the var is computed. */
     644                 :      32676 :   retval = find_retval (return_bb);
     645                 :      32676 :   if (!retval)
     646                 :            :     {
     647                 :            :       /* If there is a return_bb with no return value in function returning
     648                 :            :          value by reference, also make the split part return void, otherwise
     649                 :            :          we expansion would try to create a non-POD temporary, which is
     650                 :            :          invalid.  */
     651                 :      16330 :       if (return_bb != EXIT_BLOCK_PTR_FOR_FN (cfun)
     652                 :      11693 :           && DECL_RESULT (current_function_decl)
     653                 :      39716 :           && DECL_BY_REFERENCE (DECL_RESULT (current_function_decl)))
     654                 :          0 :         current->split_part_set_retval = false;
     655                 :            :       else
     656                 :      16330 :         current->split_part_set_retval = true;
     657                 :            :     }
     658                 :      16346 :   else if (is_gimple_min_invariant (retval))
     659                 :       1429 :     current->split_part_set_retval = false;
     660                 :            :   /* Special case is value returned by reference we record as if it was non-ssa
     661                 :            :      set to result_decl.  */
     662                 :      14917 :   else if (TREE_CODE (retval) == SSA_NAME
     663                 :      14253 :            && SSA_NAME_VAR (retval)
     664                 :       4098 :            && TREE_CODE (SSA_NAME_VAR (retval)) == RESULT_DECL
     665                 :      14917 :            && DECL_BY_REFERENCE (DECL_RESULT (current_function_decl)))
     666                 :          0 :     current->split_part_set_retval
     667                 :          0 :        = bitmap_bit_p (non_ssa_vars, DECL_UID (SSA_NAME_VAR (retval)));
     668                 :      14917 :   else if (TREE_CODE (retval) == SSA_NAME)
     669                 :      14253 :     current->split_part_set_retval
     670                 :      14253 :       = split_part_set_ssa_name_p (retval, current, return_bb);
     671                 :        664 :   else if (TREE_CODE (retval) == PARM_DECL)
     672                 :          0 :     current->split_part_set_retval = false;
     673                 :        664 :   else if (VAR_P (retval)
     674                 :        480 :            || TREE_CODE (retval) == RESULT_DECL)
     675                 :        664 :     current->split_part_set_retval
     676                 :        664 :       = bitmap_bit_p (non_ssa_vars, DECL_UID (retval));
     677                 :            :   else
     678                 :          0 :     current->split_part_set_retval = true;
     679                 :            : 
     680                 :            :   /* split_function fixes up at most one PHI non-virtual PHI node in return_bb,
     681                 :            :      for the return value.  If there are other PHIs, give up.  */
     682                 :      32676 :   if (return_bb != EXIT_BLOCK_PTR_FOR_FN (cfun))
     683                 :            :     {
     684                 :      28039 :       gphi_iterator psi;
     685                 :            : 
     686                 :      60958 :       for (psi = gsi_start_phis (return_bb); !gsi_end_p (psi); gsi_next (&psi))
     687                 :      32919 :         if (!virtual_operand_p (gimple_phi_result (psi.phi ()))
     688                 :      43968 :             && !(retval
     689                 :      11049 :                  && current->split_part_set_retval
     690                 :      11049 :                  && TREE_CODE (retval) == SSA_NAME
     691                 :      22098 :                  && !DECL_BY_REFERENCE (DECL_RESULT (current_function_decl))
     692                 :      11049 :                  && SSA_NAME_DEF_STMT (retval) == psi.phi ()))
     693                 :            :           {
     694                 :          0 :             if (dump_file && (dump_flags & TDF_DETAILS))
     695                 :          0 :               fprintf (dump_file,
     696                 :            :                        "  Refused: return bb has extra PHIs\n");
     697                 :          0 :             return;
     698                 :            :           }
     699                 :            :     }
     700                 :            : 
     701                 :      32676 :   if (dump_file && (dump_flags & TDF_DETAILS))
     702                 :          2 :     fprintf (dump_file, "  Accepted!\n");
     703                 :            : 
     704                 :            :   /* At the moment chose split point with lowest count and that leaves
     705                 :            :      out smallest size of header.
     706                 :            :      In future we might re-consider this heuristics.  */
     707                 :      32676 :   if (!best_split_point.split_bbs
     708                 :       9662 :       || best_split_point.count
     709                 :       9662 :          > current->count
     710                 :      39162 :       || (best_split_point.count == current->count 
     711                 :       2237 :           && best_split_point.split_size < current->split_size))
     712                 :            :         
     713                 :            :     {
     714                 :      27279 :       if (dump_file && (dump_flags & TDF_DETAILS))
     715                 :          1 :         fprintf (dump_file, "  New best split point!\n");
     716                 :      27279 :       if (best_split_point.ssa_names_to_pass)
     717                 :            :         {
     718                 :       4265 :           BITMAP_FREE (best_split_point.ssa_names_to_pass);
     719                 :       4265 :           BITMAP_FREE (best_split_point.split_bbs);
     720                 :            :         }
     721                 :      27279 :       best_split_point = *current;
     722                 :      27279 :       best_split_point.ssa_names_to_pass = BITMAP_ALLOC (NULL);
     723                 :      27279 :       bitmap_copy (best_split_point.ssa_names_to_pass,
     724                 :      27279 :                    current->ssa_names_to_pass);
     725                 :      27279 :       best_split_point.split_bbs = BITMAP_ALLOC (NULL);
     726                 :      27279 :       bitmap_copy (best_split_point.split_bbs, current->split_bbs);
     727                 :            :     }
     728                 :            : }
     729                 :            : 
     730                 :            : /* Return basic block containing RETURN statement.  We allow basic blocks
     731                 :            :    of the form:
     732                 :            :    <retval> = tmp_var;
     733                 :            :    return <retval>
     734                 :            :    but return_bb cannot be more complex than this (except for
     735                 :            :    -fsanitize=thread we allow TSAN_FUNC_EXIT () internal call in there).
     736                 :            :    If nothing is found, return the exit block.
     737                 :            : 
     738                 :            :    When there are multiple RETURN statement, chose one with return value,
     739                 :            :    since that one is more likely shared by multiple code paths.
     740                 :            : 
     741                 :            :    Return BB is special, because for function splitting it is the only
     742                 :            :    basic block that is duplicated in between header and split part of the
     743                 :            :    function.
     744                 :            : 
     745                 :            :    TODO: We might support multiple return blocks.  */
     746                 :            : 
     747                 :            : static basic_block
     748                 :     926883 : find_return_bb (void)
     749                 :            : {
     750                 :     926883 :   edge e;
     751                 :     926883 :   basic_block return_bb = EXIT_BLOCK_PTR_FOR_FN (cfun);
     752                 :     926883 :   gimple_stmt_iterator bsi;
     753                 :     926883 :   bool found_return = false;
     754                 :     926883 :   tree retval = NULL_TREE;
     755                 :            : 
     756                 :     926883 :   if (!single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
     757                 :            :     return return_bb;
     758                 :            : 
     759                 :     926709 :   e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
     760                 :    8232560 :   for (bsi = gsi_last_bb (e->src); !gsi_end_p (bsi); gsi_prev (&bsi))
     761                 :            :     {
     762                 :    3753920 :       gimple *stmt = gsi_stmt (bsi);
     763                 :    3753920 :       if (gimple_code (stmt) == GIMPLE_LABEL
     764                 :    3749540 :           || is_gimple_debug (stmt)
     765                 :    5443320 :           || gimple_clobber_p (stmt))
     766                 :            :         ;
     767                 :    1496740 :       else if (gimple_code (stmt) == GIMPLE_ASSIGN
     768                 :     396774 :                && found_return
     769                 :    1800400 :                && gimple_assign_single_p (stmt)
     770                 :     309247 :                && (auto_var_in_fn_p (gimple_assign_rhs1 (stmt),
     771                 :            :                                      current_function_decl)
     772                 :     301909 :                    || is_gimple_min_invariant (gimple_assign_rhs1 (stmt)))
     773                 :    1569780 :                && retval == gimple_assign_lhs (stmt))
     774                 :            :         ;
     775                 :    1491160 :       else if (greturn *return_stmt = dyn_cast <greturn *> (stmt))
     776                 :            :         {
     777                 :     926709 :           found_return = true;
     778                 :     926709 :           retval = gimple_return_retval (return_stmt);
     779                 :            :         }
     780                 :            :       /* For -fsanitize=thread, allow also TSAN_FUNC_EXIT () in the return
     781                 :            :          bb.  */
     782                 :     564448 :       else if ((flag_sanitize & SANITIZE_THREAD)
     783                 :     564448 :                && gimple_call_internal_p (stmt, IFN_TSAN_FUNC_EXIT))
     784                 :            :         ;
     785                 :            :       else
     786                 :            :         break;
     787                 :            :     }
     788                 :     926709 :   if (gsi_end_p (bsi) && found_return)
     789                 :     362361 :     return_bb = e->src;
     790                 :            : 
     791                 :            :   return return_bb;
     792                 :            : }
     793                 :            : 
     794                 :            : /* Given return basic block RETURN_BB, see where return value is really
     795                 :            :    stored.  */
     796                 :            : static tree
     797                 :      49251 : find_retval (basic_block return_bb)
     798                 :            : {
     799                 :      49251 :   gimple_stmt_iterator bsi;
     800                 :     190565 :   for (bsi = gsi_start_bb (return_bb); !gsi_end_p (bsi); gsi_next (&bsi))
     801                 :     136677 :     if (greturn *return_stmt = dyn_cast <greturn *> (gsi_stmt (bsi)))
     802                 :      44596 :       return gimple_return_retval (return_stmt);
     803                 :      92081 :     else if (gimple_code (gsi_stmt (bsi)) == GIMPLE_ASSIGN
     804                 :      92081 :              && !gimple_clobber_p (gsi_stmt (bsi)))
     805                 :         18 :       return gimple_assign_rhs1 (gsi_stmt (bsi));
     806                 :            :   return NULL;
     807                 :            : }
     808                 :            : 
     809                 :            : /* Callback for walk_stmt_load_store_addr_ops.  If T is non-SSA automatic
     810                 :            :    variable, mark it as used in bitmap passed via DATA.
     811                 :            :    Return true when access to T prevents splitting the function.  */
     812                 :            : 
     813                 :            : static bool
     814                 :    3922570 : mark_nonssa_use (gimple *, tree t, tree, void *data)
     815                 :            : {
     816                 :    3922570 :   t = get_base_address (t);
     817                 :            : 
     818                 :    3922570 :   if (!t || is_gimple_reg (t))
     819                 :          0 :     return false;
     820                 :            : 
     821                 :            :   /* At present we can't pass non-SSA arguments to split function.
     822                 :            :      FIXME: this can be relaxed by passing references to arguments.  */
     823                 :    3922570 :   if (TREE_CODE (t) == PARM_DECL)
     824                 :            :     {
     825                 :     123591 :       if (dump_file && (dump_flags & TDF_DETAILS))
     826                 :          0 :         fprintf (dump_file,
     827                 :            :                  "Cannot split: use of non-ssa function parameter.\n");
     828                 :     123591 :       return true;
     829                 :            :     }
     830                 :            : 
     831                 :    1406960 :   if ((VAR_P (t) && auto_var_in_fn_p (t, current_function_decl))
     832                 :    2668400 :       || TREE_CODE (t) == RESULT_DECL
     833                 :    6455500 :       || (TREE_CODE (t) == LABEL_DECL && FORCED_LABEL (t)))
     834                 :    1142500 :     bitmap_set_bit ((bitmap)data, DECL_UID (t));
     835                 :            : 
     836                 :            :   /* For DECL_BY_REFERENCE, the return value is actually a pointer.  We want
     837                 :            :      to pretend that the value pointed to is actual result decl.  */
     838                 :    3798980 :   if ((TREE_CODE (t) == MEM_REF || INDIRECT_REF_P (t))
     839                 :    2034910 :       && TREE_CODE (TREE_OPERAND (t, 0)) == SSA_NAME
     840                 :    2034350 :       && SSA_NAME_VAR (TREE_OPERAND (t, 0))
     841                 :    1674160 :       && TREE_CODE (SSA_NAME_VAR (TREE_OPERAND (t, 0))) == RESULT_DECL
     842                 :    3871430 :       && DECL_BY_REFERENCE (DECL_RESULT (current_function_decl)))
     843                 :      36225 :     return
     844                 :      36225 :       bitmap_bit_p ((bitmap)data,
     845                 :      36225 :                     DECL_UID (DECL_RESULT (current_function_decl)));
     846                 :            : 
     847                 :            :   return false;
     848                 :            : }
     849                 :            : 
     850                 :            : /* Compute local properties of basic block BB we collect when looking for
     851                 :            :    split points.  We look for ssa defs and store them in SET_SSA_NAMES,
     852                 :            :    for ssa uses and store them in USED_SSA_NAMES and for any non-SSA automatic
     853                 :            :    vars stored in NON_SSA_VARS.
     854                 :            : 
     855                 :            :    When BB has edge to RETURN_BB, collect uses in RETURN_BB too.  
     856                 :            : 
     857                 :            :    Return false when BB contains something that prevents it from being put into
     858                 :            :    split function.  */
     859                 :            : 
     860                 :            : static bool
     861                 :    2182600 : visit_bb (basic_block bb, basic_block return_bb,
     862                 :            :           bitmap set_ssa_names, bitmap used_ssa_names,
     863                 :            :           bitmap non_ssa_vars)
     864                 :            : {
     865                 :    2182600 :   edge e;
     866                 :    2182600 :   edge_iterator ei;
     867                 :    2182600 :   bool can_split = true;
     868                 :            : 
     869                 :   29138500 :   for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi);
     870                 :   24773300 :        gsi_next (&bsi))
     871                 :            :     {
     872                 :   24773300 :       gimple *stmt = gsi_stmt (bsi);
     873                 :   24773300 :       tree op;
     874                 :   24773300 :       ssa_op_iter iter;
     875                 :   24773300 :       tree decl;
     876                 :            : 
     877                 :   24773300 :       if (is_gimple_debug (stmt))
     878                 :   17889000 :         continue;
     879                 :            : 
     880                 :    7536350 :       if (gimple_clobber_p (stmt))
     881                 :     652076 :         continue;
     882                 :            : 
     883                 :            :       /* FIXME: We can split regions containing EH.  We cannot however
     884                 :            :          split RESX, EH_DISPATCH and EH_POINTER referring to same region
     885                 :            :          into different partitions.  This would require tracking of
     886                 :            :          EH regions and checking in consider_split_point if they 
     887                 :            :          are not used elsewhere.  */
     888                 :    6884270 :       if (gimple_code (stmt) == GIMPLE_RESX)
     889                 :            :         {
     890                 :      67328 :           if (dump_file && (dump_flags & TDF_DETAILS))
     891                 :          0 :             fprintf (dump_file, "Cannot split: resx.\n");
     892                 :            :           can_split = false;
     893                 :            :         }
     894                 :    6884270 :       if (gimple_code (stmt) == GIMPLE_EH_DISPATCH)
     895                 :            :         {
     896                 :       7095 :           if (dump_file && (dump_flags & TDF_DETAILS))
     897                 :          0 :             fprintf (dump_file, "Cannot split: eh dispatch.\n");
     898                 :            :           can_split = false;
     899                 :            :         }
     900                 :            : 
     901                 :            :       /* Check builtins that prevent splitting.  */
     902                 :    6884270 :       if (gimple_code (stmt) == GIMPLE_CALL
     903                 :    1112990 :           && (decl = gimple_call_fndecl (stmt)) != NULL_TREE
     904                 :    7951080 :           && fndecl_built_in_p (decl, BUILT_IN_NORMAL))
     905                 :     236326 :         switch (DECL_FUNCTION_CODE (decl))
     906                 :            :           {
     907                 :            :           /* FIXME: once we will allow passing non-parm values to split part,
     908                 :            :              we need to be sure to handle correct builtin_stack_save and
     909                 :            :              builtin_stack_restore.  At the moment we are safe; there is no
     910                 :            :              way to store builtin_stack_save result in non-SSA variable
     911                 :            :              since all calls to those are compiler generated.  */
     912                 :         10 :           case BUILT_IN_APPLY:
     913                 :         10 :           case BUILT_IN_APPLY_ARGS:
     914                 :         10 :           case BUILT_IN_VA_START:
     915                 :         10 :             if (dump_file && (dump_flags & TDF_DETAILS))
     916                 :          0 :               fprintf (dump_file,
     917                 :            :                        "Cannot split: builtin_apply and va_start.\n");
     918                 :            :             can_split = false;
     919                 :            :             break;
     920                 :       7978 :           case BUILT_IN_EH_POINTER:
     921                 :       7978 :             if (dump_file && (dump_flags & TDF_DETAILS))
     922                 :          0 :               fprintf (dump_file, "Cannot split: builtin_eh_pointer.\n");
     923                 :            :             can_split = false;
     924                 :            :             break;
     925                 :            :           default:
     926                 :            :             break;
     927                 :            :           }
     928                 :            : 
     929                 :   10256600 :       FOR_EACH_SSA_TREE_OPERAND (op, stmt, iter, SSA_OP_DEF)
     930                 :    3372350 :         bitmap_set_bit (set_ssa_names, SSA_NAME_VERSION (op));
     931                 :   13907200 :       FOR_EACH_SSA_TREE_OPERAND (op, stmt, iter, SSA_OP_USE)
     932                 :    7022960 :         bitmap_set_bit (used_ssa_names, SSA_NAME_VERSION (op));
     933                 :    6884270 :       can_split &= !walk_stmt_load_store_addr_ops (stmt, non_ssa_vars,
     934                 :            :                                                    mark_nonssa_use,
     935                 :            :                                                    mark_nonssa_use,
     936                 :            :                                                    mark_nonssa_use);
     937                 :            :     }
     938                 :    2731780 :   for (gphi_iterator bsi = gsi_start_phis (bb); !gsi_end_p (bsi);
     939                 :     549178 :        gsi_next (&bsi))
     940                 :            :     {
     941                 :     549178 :       gphi *stmt = bsi.phi ();
     942                 :     549178 :       unsigned int i;
     943                 :            : 
     944                 :    1098360 :       if (virtual_operand_p (gimple_phi_result (stmt)))
     945                 :     235584 :         continue;
     946                 :     627188 :       bitmap_set_bit (set_ssa_names,
     947                 :     313594 :                       SSA_NAME_VERSION (gimple_phi_result (stmt)));
     948                 :     960149 :       for (i = 0; i < gimple_phi_num_args (stmt); i++)
     949                 :            :         {
     950                 :     646555 :           tree op = gimple_phi_arg_def (stmt, i);
     951                 :     646555 :           if (TREE_CODE (op) == SSA_NAME)
     952                 :     490680 :             bitmap_set_bit (used_ssa_names, SSA_NAME_VERSION (op));
     953                 :            :         }
     954                 :     313594 :       can_split &= !walk_stmt_load_store_addr_ops (stmt, non_ssa_vars,
     955                 :            :                                                    mark_nonssa_use,
     956                 :            :                                                    mark_nonssa_use,
     957                 :            :                                                    mark_nonssa_use);
     958                 :            :     }
     959                 :            :   /* Record also uses coming from PHI operand in return BB.  */
     960                 :    5065590 :   FOR_EACH_EDGE (e, ei, bb->succs)
     961                 :    2882980 :     if (e->dest == return_bb)
     962                 :            :       {
     963                 :     909279 :         for (gphi_iterator bsi = gsi_start_phis (return_bb);
     964                 :    1282740 :              !gsi_end_p (bsi);
     965                 :     373464 :              gsi_next (&bsi))
     966                 :            :           {
     967                 :     373464 :             gphi *stmt = bsi.phi ();
     968                 :     373464 :             tree op = gimple_phi_arg_def (stmt, e->dest_idx);
     969                 :            : 
     970                 :     746928 :             if (virtual_operand_p (gimple_phi_result (stmt)))
     971                 :     230830 :               continue;
     972                 :     142634 :             if (TREE_CODE (op) == SSA_NAME)
     973                 :      65168 :               bitmap_set_bit (used_ssa_names, SSA_NAME_VERSION (op));
     974                 :            :             else
     975                 :      77466 :               can_split &= !mark_nonssa_use (stmt, op, op, non_ssa_vars);
     976                 :            :           }
     977                 :            :       }
     978                 :    2182600 :   return can_split;
     979                 :            : }
     980                 :            : 
     981                 :            : /* Stack entry for recursive DFS walk in find_split_point.  */
     982                 :            : 
     983                 :    3109470 : class stack_entry
     984                 :            : {
     985                 :            : public:
     986                 :            :   /* Basic block we are examining.  */
     987                 :            :   basic_block bb;
     988                 :            : 
     989                 :            :   /* SSA names set and used by the BB and all BBs reachable
     990                 :            :      from it via DFS walk.  */
     991                 :            :   bitmap set_ssa_names, used_ssa_names;
     992                 :            :   bitmap non_ssa_vars;
     993                 :            : 
     994                 :            :   /* All BBS visited from this BB via DFS walk.  */
     995                 :            :   bitmap bbs_visited;
     996                 :            : 
     997                 :            :   /* Last examined edge in DFS walk.  Since we walk unoriented graph,
     998                 :            :      the value is up to sum of incoming and outgoing edges of BB.  */
     999                 :            :   unsigned int edge_num;
    1000                 :            : 
    1001                 :            :   /* Stack entry index of earliest BB reachable from current BB
    1002                 :            :      or any BB visited later in DFS walk.  */
    1003                 :            :   int earliest;
    1004                 :            : 
    1005                 :            :   /* Overall time and size of all BBs reached from this BB in DFS walk.  */
    1006                 :            :   sreal overall_time;
    1007                 :            :   int overall_size;
    1008                 :            : 
    1009                 :            :   /* When false we cannot split on this BB.  */
    1010                 :            :   bool can_split;
    1011                 :            : };
    1012                 :            : 
    1013                 :            : 
    1014                 :            : /* Find all articulations and call consider_split on them.
    1015                 :            :    OVERALL_TIME and OVERALL_SIZE is time and size of the function.
    1016                 :            : 
    1017                 :            :    We perform basic algorithm for finding an articulation in a graph
    1018                 :            :    created from CFG by considering it to be an unoriented graph.
    1019                 :            : 
    1020                 :            :    The articulation is discovered via DFS walk. We collect earliest
    1021                 :            :    basic block on stack that is reachable via backward edge.  Articulation
    1022                 :            :    is any basic block such that there is no backward edge bypassing it.
    1023                 :            :    To reduce stack usage we maintain heap allocated stack in STACK vector.
    1024                 :            :    AUX pointer of BB is set to index it appears in the stack or -1 once
    1025                 :            :    it is visited and popped off the stack.
    1026                 :            : 
    1027                 :            :    The algorithm finds articulation after visiting the whole component
    1028                 :            :    reachable by it.  This makes it convenient to collect information about
    1029                 :            :    the component used by consider_split.  */
    1030                 :            : 
    1031                 :            : static void
    1032                 :     926867 : find_split_points (basic_block return_bb, sreal overall_time, int overall_size)
    1033                 :            : {
    1034                 :     926867 :   stack_entry first;
    1035                 :     926867 :   vec<stack_entry> stack = vNULL;
    1036                 :     926867 :   basic_block bb;
    1037                 :     926867 :   class split_point current;
    1038                 :            : 
    1039                 :     926867 :   current.header_time = overall_time;
    1040                 :     926867 :   current.header_size = overall_size;
    1041                 :     926867 :   current.split_time = 0;
    1042                 :     926867 :   current.split_size = 0;
    1043                 :     926867 :   current.ssa_names_to_pass = BITMAP_ALLOC (NULL);
    1044                 :            : 
    1045                 :     926867 :   first.bb = ENTRY_BLOCK_PTR_FOR_FN (cfun);
    1046                 :     926867 :   first.edge_num = 0;
    1047                 :     926867 :   first.overall_time = 0;
    1048                 :     926867 :   first.overall_size = 0;
    1049                 :     926867 :   first.earliest = INT_MAX;
    1050                 :     926867 :   first.set_ssa_names = 0;
    1051                 :     926867 :   first.used_ssa_names = 0;
    1052                 :     926867 :   first.non_ssa_vars = 0;
    1053                 :     926867 :   first.bbs_visited = 0;
    1054                 :     926867 :   first.can_split = false;
    1055                 :     926867 :   stack.safe_push (first);
    1056                 :     926867 :   ENTRY_BLOCK_PTR_FOR_FN (cfun)->aux = (void *)(intptr_t)-1;
    1057                 :            : 
    1058                 :   10533200 :   while (!stack.is_empty ())
    1059                 :            :     {
    1060                 :    9606300 :       stack_entry *entry = &stack.last ();
    1061                 :            : 
    1062                 :            :       /* We are walking an acyclic graph, so edge_num counts
    1063                 :            :          succ and pred edges together.  However when considering
    1064                 :            :          articulation, we want to have processed everything reachable
    1065                 :            :          from articulation but nothing that reaches into it.  */
    1066                 :    9606300 :       if (entry->edge_num == EDGE_COUNT (entry->bb->succs)
    1067                 :    9606300 :           && entry->bb != ENTRY_BLOCK_PTR_FOR_FN (cfun))
    1068                 :            :         {
    1069                 :    2182600 :           int pos = stack.length ();
    1070                 :    2182600 :           entry->can_split &= visit_bb (entry->bb, return_bb,
    1071                 :            :                                         entry->set_ssa_names,
    1072                 :            :                                         entry->used_ssa_names,
    1073                 :    2182600 :                                         entry->non_ssa_vars);
    1074                 :    1448810 :           if (pos <= entry->earliest && !entry->can_split
    1075                 :    2350260 :               && dump_file && (dump_flags & TDF_DETAILS))
    1076                 :          0 :             fprintf (dump_file,
    1077                 :            :                      "found articulation at bb %i but cannot split\n",
    1078                 :          0 :                      entry->bb->index);
    1079                 :    2182600 :           if (pos <= entry->earliest && entry->can_split)
    1080                 :            :              {
    1081                 :    1281160 :                if (dump_file && (dump_flags & TDF_DETAILS))
    1082                 :          4 :                  fprintf (dump_file, "found articulation at bb %i\n",
    1083                 :          4 :                           entry->bb->index);
    1084                 :    1281160 :                current.entry_bb = entry->bb;
    1085                 :    1281160 :                current.ssa_names_to_pass = BITMAP_ALLOC (NULL);
    1086                 :    1281160 :                bitmap_and_compl (current.ssa_names_to_pass,
    1087                 :    1281160 :                                  entry->used_ssa_names, entry->set_ssa_names);
    1088                 :    1281160 :                current.header_time = overall_time - entry->overall_time;
    1089                 :    1281160 :                current.header_size = overall_size - entry->overall_size;
    1090                 :    1281160 :                current.split_time = entry->overall_time;
    1091                 :    1281160 :                current.split_size = entry->overall_size;
    1092                 :    1281160 :                current.split_bbs = entry->bbs_visited;
    1093                 :    1281160 :                consider_split (&current, entry->non_ssa_vars, return_bb);
    1094                 :    1281160 :                BITMAP_FREE (current.ssa_names_to_pass);
    1095                 :            :              }
    1096                 :            :         }
    1097                 :            :       /* Do actual DFS walk.  */
    1098                 :   19212600 :       if (entry->edge_num
    1099                 :    9606300 :           < (EDGE_COUNT (entry->bb->succs)
    1100                 :   17358900 :              + EDGE_COUNT (entry->bb->preds)))
    1101                 :            :         {
    1102                 :    6496830 :           edge e;
    1103                 :    6496830 :           basic_block dest;
    1104                 :    6496830 :           if (entry->edge_num < EDGE_COUNT (entry->bb->succs))
    1105                 :            :             {
    1106                 :    3809850 :               e = EDGE_SUCC (entry->bb, entry->edge_num);
    1107                 :    3809850 :               dest = e->dest;
    1108                 :            :             }
    1109                 :            :           else
    1110                 :            :             {
    1111                 :    2686980 :               e = EDGE_PRED (entry->bb, entry->edge_num
    1112                 :            :                              - EDGE_COUNT (entry->bb->succs));
    1113                 :    2686980 :               dest = e->src;
    1114                 :            :             }
    1115                 :            : 
    1116                 :    6496830 :           entry->edge_num++;
    1117                 :            : 
    1118                 :            :           /* New BB to visit, push it to the stack.  */
    1119                 :    6496830 :           if (dest != return_bb && dest != EXIT_BLOCK_PTR_FOR_FN (cfun)
    1120                 :    5373970 :               && !dest->aux)
    1121                 :            :             {
    1122                 :    2182600 :               stack_entry new_entry;
    1123                 :            : 
    1124                 :    2182600 :               new_entry.bb = dest;
    1125                 :    2182600 :               new_entry.edge_num = 0;
    1126                 :    2182600 :               new_entry.overall_time
    1127                 :    2182600 :                  = bb_info_vec[dest->index].time;
    1128                 :    2182600 :               new_entry.overall_size
    1129                 :    2182600 :                  = bb_info_vec[dest->index].size;
    1130                 :    2182600 :               new_entry.earliest = INT_MAX;
    1131                 :    2182600 :               new_entry.set_ssa_names = BITMAP_ALLOC (NULL);
    1132                 :    2182600 :               new_entry.used_ssa_names = BITMAP_ALLOC (NULL);
    1133                 :    2182600 :               new_entry.bbs_visited = BITMAP_ALLOC (NULL);
    1134                 :    2182600 :               new_entry.non_ssa_vars = BITMAP_ALLOC (NULL);
    1135                 :    2182600 :               new_entry.can_split = true;
    1136                 :    2182600 :               bitmap_set_bit (new_entry.bbs_visited, dest->index);
    1137                 :    2182600 :               stack.safe_push (new_entry);
    1138                 :    4365210 :               dest->aux = (void *)(intptr_t)stack.length ();
    1139                 :            :             }
    1140                 :            :           /* Back edge found, record the earliest point.  */
    1141                 :    4314230 :           else if ((intptr_t)dest->aux > 0
    1142                 :    1973710 :                    && (intptr_t)dest->aux < entry->earliest)
    1143                 :    1231080 :             entry->earliest = (intptr_t)dest->aux;
    1144                 :            :         }
    1145                 :            :       /* We are done with examining the edges.  Pop off the value from stack
    1146                 :            :          and merge stuff we accumulate during the walk.  */
    1147                 :    3109470 :       else if (entry->bb != ENTRY_BLOCK_PTR_FOR_FN (cfun))
    1148                 :            :         {
    1149                 :    4365210 :           stack_entry *prev = &stack[stack.length () - 2];
    1150                 :            : 
    1151                 :    2182600 :           entry->bb->aux = (void *)(intptr_t)-1;
    1152                 :    2182600 :           prev->can_split &= entry->can_split;
    1153                 :    2182600 :           if (prev->set_ssa_names)
    1154                 :            :             {
    1155                 :    1469320 :               bitmap_ior_into (prev->set_ssa_names, entry->set_ssa_names);
    1156                 :    1469320 :               bitmap_ior_into (prev->used_ssa_names, entry->used_ssa_names);
    1157                 :    1469320 :               bitmap_ior_into (prev->bbs_visited, entry->bbs_visited);
    1158                 :    1469320 :               bitmap_ior_into (prev->non_ssa_vars, entry->non_ssa_vars);
    1159                 :            :             }
    1160                 :    2182600 :           if (prev->earliest > entry->earliest)
    1161                 :    1314760 :             prev->earliest = entry->earliest;
    1162                 :    2182600 :           prev->overall_time += entry->overall_time;
    1163                 :    2182600 :           prev->overall_size += entry->overall_size;
    1164                 :    2182600 :           BITMAP_FREE (entry->set_ssa_names);
    1165                 :    2182600 :           BITMAP_FREE (entry->used_ssa_names);
    1166                 :    2182600 :           BITMAP_FREE (entry->bbs_visited);
    1167                 :    2182600 :           BITMAP_FREE (entry->non_ssa_vars);
    1168                 :    2182600 :           stack.pop ();
    1169                 :            :         }
    1170                 :            :       else
    1171                 :     926867 :         stack.pop ();
    1172                 :            :     }
    1173                 :     926867 :   ENTRY_BLOCK_PTR_FOR_FN (cfun)->aux = NULL;
    1174                 :    3471820 :   FOR_EACH_BB_FN (bb, cfun)
    1175                 :    2544960 :     bb->aux = NULL;
    1176                 :     926867 :   stack.release ();
    1177                 :     926867 :   BITMAP_FREE (current.ssa_names_to_pass);
    1178                 :     926867 : }
    1179                 :            : 
    1180                 :            : /* Split function at SPLIT_POINT.  */
    1181                 :            : 
    1182                 :            : static void
    1183                 :      23014 : split_function (basic_block return_bb, class split_point *split_point,
    1184                 :            :                 bool add_tsan_func_exit)
    1185                 :            : {
    1186                 :      23014 :   vec<tree> args_to_pass = vNULL;
    1187                 :      23014 :   bitmap args_to_skip;
    1188                 :      23014 :   tree parm;
    1189                 :      23014 :   int num = 0;
    1190                 :      23014 :   cgraph_node *node, *cur_node = cgraph_node::get (current_function_decl);
    1191                 :      23014 :   basic_block call_bb;
    1192                 :      23014 :   gcall *call, *tsan_func_exit_call = NULL;
    1193                 :      23014 :   edge e;
    1194                 :      23014 :   edge_iterator ei;
    1195                 :      23014 :   tree retval = NULL, real_retval = NULL;
    1196                 :      23014 :   gimple *last_stmt = NULL;
    1197                 :      23014 :   unsigned int i;
    1198                 :      23014 :   tree arg, ddef;
    1199                 :            : 
    1200                 :      23014 :   if (dump_file)
    1201                 :            :     {
    1202                 :          7 :       fprintf (dump_file, "\n\nSplitting function at:\n");
    1203                 :          7 :       dump_split_point (dump_file, split_point);
    1204                 :            :     }
    1205                 :            : 
    1206                 :      23014 :   if (cur_node->can_change_signature)
    1207                 :      20402 :     args_to_skip = BITMAP_ALLOC (NULL);
    1208                 :            :   else
    1209                 :            :     args_to_skip = NULL;
    1210                 :            : 
    1211                 :            :   /* Collect the parameters of new function and args_to_skip bitmap.  */
    1212                 :      70633 :   for (parm = DECL_ARGUMENTS (current_function_decl);
    1213                 :      70633 :        parm; parm = DECL_CHAIN (parm), num++)
    1214                 :      47619 :     if (args_to_skip
    1215                 :      47619 :         && (!is_gimple_reg (parm)
    1216                 :      41602 :             || (ddef = ssa_default_def (cfun, parm)) == NULL_TREE
    1217                 :      40713 :             || !bitmap_bit_p (split_point->ssa_names_to_pass,
    1218                 :      40713 :                               SSA_NAME_VERSION (ddef))))
    1219                 :      11155 :       bitmap_set_bit (args_to_skip, num);
    1220                 :            :     else
    1221                 :            :       {
    1222                 :            :         /* This parm might not have been used up to now, but is going to be
    1223                 :            :            used, hence register it.  */
    1224                 :      36464 :         if (is_gimple_reg (parm))
    1225                 :      36426 :           arg = get_or_create_ssa_default_def (cfun, parm);
    1226                 :            :         else
    1227                 :         38 :           arg = parm;
    1228                 :            : 
    1229                 :      36464 :         if (!useless_type_conversion_p (DECL_ARG_TYPE (parm), TREE_TYPE (arg)))
    1230                 :        812 :           arg = fold_convert (DECL_ARG_TYPE (parm), arg);
    1231                 :      36464 :         args_to_pass.safe_push (arg);
    1232                 :            :       }
    1233                 :            : 
    1234                 :            :   /* See if the split function will return.  */
    1235                 :      23014 :   bool split_part_return_p = false;
    1236                 :      83918 :   FOR_EACH_EDGE (e, ei, return_bb->preds)
    1237                 :            :     {
    1238                 :      60904 :       if (bitmap_bit_p (split_point->split_bbs, e->src->index))
    1239                 :      30980 :         split_part_return_p = true;
    1240                 :            :     }
    1241                 :            : 
    1242                 :            :   /* Add return block to what will become the split function.
    1243                 :            :      We do not return; no return block is needed.  */
    1244                 :      23014 :   if (!split_part_return_p)
    1245                 :            :     ;
    1246                 :            :   /* We have no return block, so nothing is needed.  */
    1247                 :      16652 :   else if (return_bb == EXIT_BLOCK_PTR_FOR_FN (cfun))
    1248                 :            :     ;
    1249                 :            :   /* When we do not want to return value, we need to construct
    1250                 :            :      new return block with empty return statement.
    1251                 :            :      FIXME: Once we are able to change return type, we should change function
    1252                 :            :      to return void instead of just outputting function with undefined return
    1253                 :            :      value.  For structures this affects quality of codegen.  */
    1254                 :      16575 :   else if ((retval = find_retval (return_bb))
    1255                 :      16575 :            && !split_point->split_part_set_retval)
    1256                 :            :     {
    1257                 :       2082 :       bool redirected = true;
    1258                 :       2082 :       basic_block new_return_bb = create_basic_block (NULL, 0, return_bb);
    1259                 :       2082 :       gimple_stmt_iterator gsi = gsi_start_bb (new_return_bb);
    1260                 :       2082 :       gsi_insert_after (&gsi, gimple_build_return (NULL), GSI_NEW_STMT);
    1261                 :       2082 :       new_return_bb->count = profile_count::zero ();
    1262                 :       4164 :       while (redirected)
    1263                 :            :         {
    1264                 :       4372 :           redirected = false;
    1265                 :      10663 :           FOR_EACH_EDGE (e, ei, return_bb->preds)
    1266                 :       8581 :             if (bitmap_bit_p (split_point->split_bbs, e->src->index))
    1267                 :            :               {
    1268                 :       2290 :                 new_return_bb->count += e->count ();
    1269                 :       2290 :                 redirect_edge_and_branch (e, new_return_bb);
    1270                 :       2290 :                 redirected = true;
    1271                 :       2290 :                 break;
    1272                 :            :               }
    1273                 :            :         }
    1274                 :       2082 :       e = make_single_succ_edge (new_return_bb, EXIT_BLOCK_PTR_FOR_FN (cfun), 0);
    1275                 :       2082 :       add_bb_to_loop (new_return_bb, current_loops->tree_root);
    1276                 :       2082 :       bitmap_set_bit (split_point->split_bbs, new_return_bb->index);
    1277                 :            :     }
    1278                 :            :   /* When we pass around the value, use existing return block.  */
    1279                 :            :   else
    1280                 :      14493 :     bitmap_set_bit (split_point->split_bbs, return_bb->index);
    1281                 :            : 
    1282                 :            :   /* If RETURN_BB has virtual operand PHIs, they must be removed and the
    1283                 :            :      virtual operand marked for renaming as we change the CFG in a way that
    1284                 :            :      tree-inline is not able to compensate for.
    1285                 :            : 
    1286                 :            :      Note this can happen whether or not we have a return value.  If we have
    1287                 :            :      a return value, then RETURN_BB may have PHIs for real operands too.  */
    1288                 :      23014 :   if (return_bb != EXIT_BLOCK_PTR_FOR_FN (cfun))
    1289                 :            :     {
    1290                 :      19916 :       bool phi_p = false;
    1291                 :      19916 :       for (gphi_iterator gsi = gsi_start_phis (return_bb);
    1292                 :      42826 :            !gsi_end_p (gsi);)
    1293                 :            :         {
    1294                 :      22910 :           gphi *stmt = gsi.phi ();
    1295                 :      45820 :           if (!virtual_operand_p (gimple_phi_result (stmt)))
    1296                 :            :             {
    1297                 :       7626 :               gsi_next (&gsi);
    1298                 :       7626 :               continue;
    1299                 :            :             }
    1300                 :      15284 :           mark_virtual_phi_result_for_renaming (stmt);
    1301                 :      15284 :           remove_phi_node (&gsi, true);
    1302                 :      15284 :           phi_p = true;
    1303                 :            :         }
    1304                 :            :       /* In reality we have to rename the reaching definition of the
    1305                 :            :          virtual operand at return_bb as we will eventually release it
    1306                 :            :          when we remove the code region we outlined.
    1307                 :            :          So we have to rename all immediate virtual uses of that region
    1308                 :            :          if we didn't see a PHI definition yet.  */
    1309                 :            :       /* ???  In real reality we want to set the reaching vdef of the
    1310                 :            :          entry of the SESE region as the vuse of the call and the reaching
    1311                 :            :          vdef of the exit of the SESE region as the vdef of the call.  */
    1312                 :      19916 :       if (!phi_p)
    1313                 :       4632 :         for (gimple_stmt_iterator gsi = gsi_start_bb (return_bb);
    1314                 :      22210 :              !gsi_end_p (gsi);
    1315                 :      17578 :              gsi_next (&gsi))
    1316                 :            :           {
    1317                 :      18046 :             gimple *stmt = gsi_stmt (gsi);
    1318                 :      22678 :             if (gimple_vuse (stmt))
    1319                 :            :               {
    1320                 :       4632 :                 gimple_set_vuse (stmt, NULL_TREE);
    1321                 :       4632 :                 update_stmt (stmt);
    1322                 :            :               }
    1323                 :      22678 :             if (gimple_vdef (stmt))
    1324                 :            :               break;
    1325                 :            :           }
    1326                 :            :     }
    1327                 :            : 
    1328                 :      23014 :   ipa_param_adjustments *adjustments;
    1329                 :      46028 :   bool skip_return = (!split_part_return_p
    1330                 :      23014 :                       || !split_point->split_part_set_retval);
    1331                 :            :   /* TODO: Perhaps get rid of args_to_skip entirely, after we make sure the
    1332                 :            :      debug info generation and discrepancy avoiding works well too.  */
    1333                 :      20402 :   if ((args_to_skip && !bitmap_empty_p (args_to_skip))
    1334                 :      36489 :       || skip_return)
    1335                 :            :     {
    1336                 :      11105 :       vec<ipa_adjusted_param, va_gc> *new_params = NULL;
    1337                 :      11105 :       unsigned j;
    1338                 :      34477 :       for (parm = DECL_ARGUMENTS (current_function_decl), j = 0;
    1339                 :      34477 :            parm; parm = DECL_CHAIN (parm), j++)
    1340                 :      23372 :         if (!args_to_skip || !bitmap_bit_p (args_to_skip, j))
    1341                 :            :           {
    1342                 :      12217 :             ipa_adjusted_param adj;
    1343                 :      12217 :             memset (&adj, 0, sizeof (adj));
    1344                 :      12217 :             adj.op = IPA_PARAM_OP_COPY;
    1345                 :      12217 :             adj.base_index = j;
    1346                 :      12217 :             adj.prev_clone_index = j;
    1347                 :      12217 :             vec_safe_push (new_params, adj);
    1348                 :            :           }
    1349                 :      11105 :       adjustments = new ipa_param_adjustments (new_params, j, skip_return);
    1350                 :            :     }
    1351                 :            :   else
    1352                 :            :     adjustments = NULL;
    1353                 :            : 
    1354                 :            :   /* Now create the actual clone.  */
    1355                 :      23014 :   cgraph_edge::rebuild_edges ();
    1356                 :      23014 :   node = cur_node->create_version_clone_with_body
    1357                 :      23014 :     (vNULL, NULL, adjustments,
    1358                 :            :      split_point->split_bbs, split_point->entry_bb, "part");
    1359                 :      23014 :   delete adjustments;
    1360                 :      23014 :   node->split_part = true;
    1361                 :            : 
    1362                 :      23014 :   if (cur_node->same_comdat_group)
    1363                 :            :     {
    1364                 :            :       /* TODO: call is versionable if we make sure that all
    1365                 :            :          callers are inside of a comdat group.  */
    1366                 :       1151 :       cur_node->calls_comdat_local = true;
    1367                 :       1151 :       node->add_to_same_comdat_group (cur_node);
    1368                 :            :     }
    1369                 :            : 
    1370                 :            : 
    1371                 :            :   /* Let's take a time profile for splitted function.  */
    1372                 :      23014 :   if (cur_node->tp_first_run)
    1373                 :          0 :     node->tp_first_run = cur_node->tp_first_run + 1;
    1374                 :            : 
    1375                 :            :   /* For usual cloning it is enough to clear builtin only when signature
    1376                 :            :      changes.  For partial inlining we however cannot expect the part
    1377                 :            :      of builtin implementation to have same semantic as the whole.  */
    1378                 :      23014 :   if (fndecl_built_in_p (node->decl))
    1379                 :         32 :     set_decl_built_in_function (node->decl, NOT_BUILT_IN, 0);
    1380                 :            : 
    1381                 :            :   /* If return_bb contains any clobbers that refer to SSA_NAMEs
    1382                 :            :      set in the split part, remove them.  Also reset debug stmts that
    1383                 :            :      refer to SSA_NAMEs set in the split part.  */
    1384                 :      23014 :   if (return_bb != EXIT_BLOCK_PTR_FOR_FN (cfun))
    1385                 :            :     {
    1386                 :      19916 :       gimple_stmt_iterator gsi = gsi_start_bb (return_bb);
    1387                 :      84221 :       while (!gsi_end_p (gsi))
    1388                 :            :         {
    1389                 :      64305 :           tree op;
    1390                 :      64305 :           ssa_op_iter iter;
    1391                 :      64305 :           gimple *stmt = gsi_stmt (gsi);
    1392                 :      64305 :           bool remove = false;
    1393                 :      64305 :           if (gimple_clobber_p (stmt) || is_gimple_debug (stmt))
    1394                 :      45671 :             FOR_EACH_SSA_TREE_OPERAND (op, stmt, iter, SSA_OP_USE)
    1395                 :            :               {
    1396                 :       1641 :                 basic_block bb = gimple_bb (SSA_NAME_DEF_STMT (op));
    1397                 :       1641 :                 if (op != retval
    1398                 :       1641 :                     && bb
    1399                 :         80 :                     && bb != return_bb
    1400                 :       1712 :                     && bitmap_bit_p (split_point->split_bbs, bb->index))
    1401                 :            :                   {
    1402                 :          0 :                     if (is_gimple_debug (stmt))
    1403                 :            :                       {
    1404                 :          0 :                         gimple_debug_bind_reset_value (stmt);
    1405                 :          0 :                         update_stmt (stmt);
    1406                 :            :                       }
    1407                 :            :                     else
    1408                 :            :                       remove = true;
    1409                 :            :                     break;
    1410                 :            :                   }
    1411                 :            :               }
    1412                 :          0 :           if (remove)
    1413                 :          0 :             gsi_remove (&gsi, true);
    1414                 :            :           else
    1415                 :      64305 :             gsi_next (&gsi);
    1416                 :            :         }
    1417                 :            :     }
    1418                 :            : 
    1419                 :            :   /* If the original function is declared inline, there is no point in issuing
    1420                 :            :      a warning for the non-inlinable part.  */
    1421                 :      23014 :   DECL_NO_INLINE_WARNING_P (node->decl) = 1;
    1422                 :      23014 :   cur_node->remove_callees ();
    1423                 :      23014 :   cur_node->remove_all_references ();
    1424                 :      23014 :   if (!split_part_return_p)
    1425                 :       6362 :     TREE_THIS_VOLATILE (node->decl) = 1;
    1426                 :      23014 :   if (dump_file)
    1427                 :          7 :     dump_function_to_file (node->decl, dump_file, dump_flags);
    1428                 :            : 
    1429                 :            :   /* Create the basic block we place call into.  It is the entry basic block
    1430                 :            :      split after last label.  */
    1431                 :      23014 :   call_bb = split_point->entry_bb;
    1432                 :      46386 :   for (gimple_stmt_iterator gsi = gsi_start_bb (call_bb); !gsi_end_p (gsi);)
    1433                 :      23372 :     if (gimple_code (gsi_stmt (gsi)) == GIMPLE_LABEL)
    1434                 :            :       {
    1435                 :        358 :         last_stmt = gsi_stmt (gsi);
    1436                 :        358 :         gsi_next (&gsi);
    1437                 :            :       }
    1438                 :            :     else
    1439                 :            :       break;
    1440                 :      23014 :   call_bb->count = split_point->count;
    1441                 :      23014 :   e = split_block (split_point->entry_bb, last_stmt);
    1442                 :      23014 :   remove_edge (e);
    1443                 :            : 
    1444                 :            :   /* Produce the call statement.  */
    1445                 :      23014 :   gimple_stmt_iterator gsi = gsi_last_bb (call_bb);
    1446                 :      77433 :   FOR_EACH_VEC_ELT (args_to_pass, i, arg)
    1447                 :      36464 :     if (!is_gimple_val (arg))
    1448                 :            :       {
    1449                 :        830 :         arg = force_gimple_operand_gsi (&gsi, arg, true, NULL_TREE,
    1450                 :            :                                         false, GSI_CONTINUE_LINKING);
    1451                 :        830 :         args_to_pass[i] = arg;
    1452                 :            :       }
    1453                 :      23014 :   call = gimple_build_call_vec (node->decl, args_to_pass);
    1454                 :      23014 :   gimple_set_block (call, DECL_INITIAL (current_function_decl));
    1455                 :      23014 :   args_to_pass.release ();
    1456                 :            : 
    1457                 :            :   /* For optimized away parameters, add on the caller side
    1458                 :            :      before the call
    1459                 :            :      DEBUG D#X => parm_Y(D)
    1460                 :            :      stmts and associate D#X with parm in decl_debug_args_lookup
    1461                 :            :      vector to say for debug info that if parameter parm had been passed,
    1462                 :            :      it would have value parm_Y(D).  */
    1463                 :      23014 :   if (args_to_skip)
    1464                 :            :     {
    1465                 :      20402 :       vec<tree, va_gc> **debug_args = NULL;
    1466                 :      20402 :       unsigned i = 0, len = 0;
    1467                 :      20402 :       if (MAY_HAVE_DEBUG_BIND_STMTS)
    1468                 :            :         {
    1469                 :      19047 :           debug_args = decl_debug_args_lookup (node->decl);
    1470                 :      19047 :           if (debug_args)
    1471                 :       5930 :             len = vec_safe_length (*debug_args);
    1472                 :            :         }
    1473                 :      62927 :       for (parm = DECL_ARGUMENTS (current_function_decl), num = 0;
    1474                 :      62927 :            parm; parm = DECL_CHAIN (parm), num++)
    1475                 :      42525 :         if (bitmap_bit_p (args_to_skip, num) && is_gimple_reg (parm))
    1476                 :            :           {
    1477                 :      10232 :             tree ddecl;
    1478                 :      10232 :             gimple *def_temp;
    1479                 :            : 
    1480                 :            :             /* This needs to be done even without
    1481                 :            :                MAY_HAVE_DEBUG_BIND_STMTS, otherwise if it didn't exist
    1482                 :            :                before, we'd end up with different SSA_NAME_VERSIONs
    1483                 :            :                between -g and -g0.  */
    1484                 :      10232 :             arg = get_or_create_ssa_default_def (cfun, parm);
    1485                 :      10232 :             if (!MAY_HAVE_DEBUG_BIND_STMTS || debug_args == NULL)
    1486                 :        667 :               continue;
    1487                 :            : 
    1488                 :      13200 :             while (i < len && (**debug_args)[i] != DECL_ORIGIN (parm))
    1489                 :       3635 :               i += 2;
    1490                 :       9565 :             if (i >= len)
    1491                 :          0 :               continue;
    1492                 :       9565 :             ddecl = (**debug_args)[i + 1];
    1493                 :       9565 :             def_temp
    1494                 :       9565 :               = gimple_build_debug_bind (ddecl, unshare_expr (arg), call);
    1495                 :       9565 :             gsi_insert_after (&gsi, def_temp, GSI_NEW_STMT);
    1496                 :            :           }
    1497                 :      20402 :       BITMAP_FREE (args_to_skip);
    1498                 :            :     }
    1499                 :            : 
    1500                 :            :   /* We avoid address being taken on any variable used by split part,
    1501                 :            :      so return slot optimization is always possible.  Moreover this is
    1502                 :            :      required to make DECL_BY_REFERENCE work.  */
    1503                 :      23014 :   if (aggregate_value_p (DECL_RESULT (current_function_decl),
    1504                 :      23014 :                          TREE_TYPE (current_function_decl))
    1505                 :      23014 :       && (!is_gimple_reg_type (TREE_TYPE (DECL_RESULT (current_function_decl)))
    1506                 :         86 :           || DECL_BY_REFERENCE (DECL_RESULT (current_function_decl))))
    1507                 :        380 :     gimple_call_set_return_slot_opt (call, true);
    1508                 :            : 
    1509                 :      23014 :   if (add_tsan_func_exit)
    1510                 :          2 :     tsan_func_exit_call = gimple_build_call_internal (IFN_TSAN_FUNC_EXIT, 0);
    1511                 :            : 
    1512                 :            :   /* Update return value.  This is bit tricky.  When we do not return,
    1513                 :            :      do nothing.  When we return we might need to update return_bb
    1514                 :            :      or produce a new return statement.  */
    1515                 :      23014 :   if (!split_part_return_p)
    1516                 :            :     {
    1517                 :       6362 :       gsi_insert_after (&gsi, call, GSI_NEW_STMT);
    1518                 :       6362 :       if (tsan_func_exit_call)
    1519                 :          0 :         gsi_insert_after (&gsi, tsan_func_exit_call, GSI_NEW_STMT);
    1520                 :            :     }
    1521                 :            :   else
    1522                 :            :     {
    1523                 :      33304 :       e = make_single_succ_edge (call_bb, return_bb,
    1524                 :      16652 :                                  return_bb == EXIT_BLOCK_PTR_FOR_FN (cfun)
    1525                 :      16652 :                                  ? 0 : EDGE_FALLTHRU);
    1526                 :            : 
    1527                 :            :       /* If there is return basic block, see what value we need to store
    1528                 :            :          return value into and put call just before it.  */
    1529                 :      16652 :       if (return_bb != EXIT_BLOCK_PTR_FOR_FN (cfun))
    1530                 :            :         {
    1531                 :      16575 :           real_retval = retval;
    1532                 :      16575 :           if (real_retval && split_point->split_part_set_retval)
    1533                 :            :             {
    1534                 :       8011 :               gphi_iterator psi;
    1535                 :            : 
    1536                 :            :               /* See if we need new SSA_NAME for the result.
    1537                 :            :                  When DECL_BY_REFERENCE is true, retval is actually pointer to
    1538                 :            :                  return value and it is constant in whole function.  */
    1539                 :       8011 :               if (TREE_CODE (retval) == SSA_NAME
    1540                 :      15529 :                   && !DECL_BY_REFERENCE (DECL_RESULT (current_function_decl)))
    1541                 :            :                 {
    1542                 :       7518 :                   retval = copy_ssa_name (retval, call);
    1543                 :            : 
    1544                 :            :                   /* See if there is PHI defining return value.  */
    1545                 :       7518 :                   for (psi = gsi_start_phis (return_bb);
    1546                 :       7518 :                        !gsi_end_p (psi); gsi_next (&psi))
    1547                 :      15036 :                     if (!virtual_operand_p (gimple_phi_result (psi.phi ())))
    1548                 :            :                       break;
    1549                 :            : 
    1550                 :            :                   /* When there is PHI, just update its value.  */
    1551                 :       7518 :                   if (TREE_CODE (retval) == SSA_NAME
    1552                 :       7518 :                       && !gsi_end_p (psi))
    1553                 :       7518 :                     add_phi_arg (psi.phi (), retval, e, UNKNOWN_LOCATION);
    1554                 :            :                   /* Otherwise update the return BB itself.
    1555                 :            :                      find_return_bb allows at most one assignment to return value,
    1556                 :            :                      so update first statement.  */
    1557                 :            :                   else
    1558                 :            :                     {
    1559                 :          0 :                       gimple_stmt_iterator bsi;
    1560                 :          0 :                       for (bsi = gsi_start_bb (return_bb); !gsi_end_p (bsi);
    1561                 :          0 :                            gsi_next (&bsi))
    1562                 :          0 :                         if (greturn *return_stmt
    1563                 :          0 :                               = dyn_cast <greturn *> (gsi_stmt (bsi)))
    1564                 :            :                           {
    1565                 :          0 :                             gimple_return_set_retval (return_stmt, retval);
    1566                 :            :                             break;
    1567                 :            :                           }
    1568                 :          0 :                         else if (gimple_code (gsi_stmt (bsi)) == GIMPLE_ASSIGN
    1569                 :          0 :                                  && !gimple_clobber_p (gsi_stmt (bsi)))
    1570                 :            :                           {
    1571                 :          0 :                             gimple_assign_set_rhs1 (gsi_stmt (bsi), retval);
    1572                 :            :                             break;
    1573                 :            :                           }
    1574                 :          0 :                       update_stmt (gsi_stmt (bsi));
    1575                 :            :                       /* Also adjust clobbers and debug stmts in return_bb.  */
    1576                 :          0 :                       for (bsi = gsi_start_bb (return_bb); !gsi_end_p (bsi);
    1577                 :          0 :                            gsi_next (&bsi))
    1578                 :            :                         {
    1579                 :          0 :                           gimple *stmt = gsi_stmt (bsi);
    1580                 :          0 :                           if (gimple_clobber_p (stmt)
    1581                 :          0 :                               || is_gimple_debug (stmt))
    1582                 :            :                             {
    1583                 :          0 :                               ssa_op_iter iter;
    1584                 :          0 :                               use_operand_p use_p;
    1585                 :          0 :                               bool update = false;
    1586                 :          0 :                               FOR_EACH_SSA_USE_OPERAND (use_p, stmt, iter,
    1587                 :            :                                                         SSA_OP_USE)
    1588                 :          0 :                                 if (USE_FROM_PTR (use_p) == real_retval)
    1589                 :            :                                   {
    1590                 :          0 :                                     SET_USE (use_p, retval);
    1591                 :          0 :                                     update = true;
    1592                 :            :                                   }
    1593                 :          0 :                               if (update)
    1594                 :          0 :                                 update_stmt (stmt);
    1595                 :            :                             }
    1596                 :            :                         }
    1597                 :            :                     }
    1598                 :            :                 }
    1599                 :      16022 :               if (DECL_BY_REFERENCE (DECL_RESULT (current_function_decl)))
    1600                 :            :                 {
    1601                 :          0 :                   gimple_call_set_lhs (call, build_simple_mem_ref (retval));
    1602                 :          0 :                   gsi_insert_after (&gsi, call, GSI_NEW_STMT);
    1603                 :            :                 }
    1604                 :            :               else
    1605                 :            :                 {
    1606                 :       8011 :                   tree restype;
    1607                 :       8011 :                   restype = TREE_TYPE (DECL_RESULT (current_function_decl));
    1608                 :       8011 :                   gsi_insert_after (&gsi, call, GSI_NEW_STMT);
    1609                 :       8011 :                   if (!useless_type_conversion_p (TREE_TYPE (retval), restype))
    1610                 :            :                     {
    1611                 :          0 :                       gimple *cpy;
    1612                 :          0 :                       tree tem = create_tmp_reg (restype);
    1613                 :          0 :                       tem = make_ssa_name (tem, call);
    1614                 :          0 :                       cpy = gimple_build_assign (retval, NOP_EXPR, tem);
    1615                 :          0 :                       gsi_insert_after (&gsi, cpy, GSI_NEW_STMT);
    1616                 :          0 :                       retval = tem;
    1617                 :            :                     }
    1618                 :       8011 :                   gimple_call_set_lhs (call, retval);
    1619                 :       8011 :                   update_stmt (call);
    1620                 :       8011 :                 }
    1621                 :            :             }
    1622                 :            :           else
    1623                 :       8564 :             gsi_insert_after (&gsi, call, GSI_NEW_STMT);
    1624                 :      16575 :           if (tsan_func_exit_call)
    1625                 :          2 :             gsi_insert_after (&gsi, tsan_func_exit_call, GSI_NEW_STMT);
    1626                 :            :         }
    1627                 :            :       /* We don't use return block (there is either no return in function or
    1628                 :            :          multiple of them).  So create new basic block with return statement.
    1629                 :            :          */
    1630                 :            :       else
    1631                 :            :         {
    1632                 :         77 :           greturn *ret;
    1633                 :         77 :           if (split_point->split_part_set_retval
    1634                 :         77 :               && !VOID_TYPE_P (TREE_TYPE (TREE_TYPE (current_function_decl))))
    1635                 :            :             {
    1636                 :         68 :               retval = DECL_RESULT (current_function_decl);
    1637                 :            : 
    1638                 :            :               /* We use temporary register to hold value when aggregate_value_p
    1639                 :            :                  is false.  Similarly for DECL_BY_REFERENCE we must avoid extra
    1640                 :            :                  copy.  */
    1641                 :         68 :               if (!aggregate_value_p (retval, TREE_TYPE (current_function_decl))
    1642                 :        136 :                   && !DECL_BY_REFERENCE (retval))
    1643                 :         68 :                 retval = create_tmp_reg (TREE_TYPE (retval));
    1644                 :         68 :               if (is_gimple_reg (retval))
    1645                 :            :                 {
    1646                 :            :                   /* When returning by reference, there is only one SSA name
    1647                 :            :                      assigned to RESULT_DECL (that is pointer to return value).
    1648                 :            :                      Look it up or create new one if it is missing.  */
    1649                 :         68 :                   if (DECL_BY_REFERENCE (retval))
    1650                 :          0 :                     retval = get_or_create_ssa_default_def (cfun, retval);
    1651                 :            :                   /* Otherwise produce new SSA name for return value.  */
    1652                 :            :                   else
    1653                 :         68 :                     retval = make_ssa_name (retval, call);
    1654                 :            :                 }
    1655                 :        136 :               if (DECL_BY_REFERENCE (DECL_RESULT (current_function_decl)))
    1656                 :          0 :                 gimple_call_set_lhs (call, build_simple_mem_ref (retval));
    1657                 :            :               else
    1658                 :         68 :                 gimple_call_set_lhs (call, retval);
    1659                 :         68 :               gsi_insert_after (&gsi, call, GSI_NEW_STMT);
    1660                 :            :             }
    1661                 :            :           else
    1662                 :            :             {
    1663                 :          9 :               gsi_insert_after (&gsi, call, GSI_NEW_STMT);
    1664                 :          9 :               if (retval
    1665                 :          0 :                   && is_gimple_reg_type (TREE_TYPE (retval))
    1666                 :          9 :                   && !is_gimple_val (retval))
    1667                 :            :                 {
    1668                 :          0 :                   gassign *g
    1669                 :          0 :                     = gimple_build_assign (make_ssa_name (TREE_TYPE (retval)),
    1670                 :            :                                            retval);
    1671                 :          0 :                   retval = gimple_assign_lhs (g);
    1672                 :          0 :                   gsi_insert_after (&gsi, g, GSI_NEW_STMT);
    1673                 :            :                 }
    1674                 :            :             }
    1675                 :         77 :           if (tsan_func_exit_call)
    1676                 :          0 :             gsi_insert_after (&gsi, tsan_func_exit_call, GSI_NEW_STMT);
    1677                 :         77 :           ret = gimple_build_return (retval);
    1678                 :         77 :           gsi_insert_after (&gsi, ret, GSI_NEW_STMT);
    1679                 :            :         }
    1680                 :            :     }
    1681                 :      23014 :   free_dominance_info (CDI_DOMINATORS);
    1682                 :      23014 :   free_dominance_info (CDI_POST_DOMINATORS);
    1683                 :      23014 :   compute_fn_summary (node, true);
    1684                 :      23014 : }
    1685                 :            : 
    1686                 :            : /* Execute function splitting pass.  */
    1687                 :            : 
    1688                 :            : static unsigned int
    1689                 :    1341850 : execute_split_functions (void)
    1690                 :            : {
    1691                 :    1341850 :   gimple_stmt_iterator bsi;
    1692                 :    1341850 :   basic_block bb;
    1693                 :    1341850 :   sreal overall_time = 0;
    1694                 :    1341850 :   int overall_size = 0;
    1695                 :    1341850 :   int todo = 0;
    1696                 :    1341850 :   struct cgraph_node *node = cgraph_node::get (current_function_decl);
    1697                 :            : 
    1698                 :    1341850 :   if (flags_from_decl_or_type (current_function_decl)
    1699                 :    1341850 :       & (ECF_NORETURN|ECF_MALLOC))
    1700                 :            :     {
    1701                 :      28994 :       if (dump_file)
    1702                 :          0 :         fprintf (dump_file, "Not splitting: noreturn/malloc function.\n");
    1703                 :      28994 :       return 0;
    1704                 :            :     }
    1705                 :    1312850 :   if (MAIN_NAME_P (DECL_NAME (current_function_decl)))
    1706                 :            :     {
    1707                 :      44260 :       if (dump_file)
    1708                 :          3 :         fprintf (dump_file, "Not splitting: main function.\n");
    1709                 :      44260 :       return 0;
    1710                 :            :     }
    1711                 :    1268590 :   if (node->frequency == NODE_FREQUENCY_UNLIKELY_EXECUTED)
    1712                 :            :     {
    1713                 :        446 :       if (dump_file)
    1714                 :          0 :         fprintf (dump_file, "Not splitting: function is unlikely executed.\n");
    1715                 :        446 :       return 0;
    1716                 :            :     }
    1717                 :            :   /* This can be relaxed; function might become inlinable after splitting
    1718                 :            :      away the uninlinable part.  */
    1719                 :    1268150 :   if (ipa_fn_summaries
    1720                 :    1268150 :       && ipa_fn_summaries->get (node)
    1721                 :    1268130 :       && !ipa_fn_summaries->get (node)->inlinable)
    1722                 :            :     {
    1723                 :     140870 :       if (dump_file)
    1724                 :          0 :         fprintf (dump_file, "Not splitting: not inlinable.\n");
    1725                 :     140870 :       return 0;
    1726                 :            :     }
    1727                 :    1127280 :   if (DECL_DISREGARD_INLINE_LIMITS (node->decl))
    1728                 :            :     {
    1729                 :      55883 :       if (dump_file)
    1730                 :          0 :         fprintf (dump_file, "Not splitting: disregarding inline limits.\n");
    1731                 :      55883 :       return 0;
    1732                 :            :     }
    1733                 :            :   /* This can be relaxed; most of versioning tests actually prevents
    1734                 :            :      a duplication.  */
    1735                 :    1071400 :   if (!tree_versionable_function_p (current_function_decl))
    1736                 :            :     {
    1737                 :         14 :       if (dump_file)
    1738                 :          0 :         fprintf (dump_file, "Not splitting: not versionable.\n");
    1739                 :         14 :       return 0;
    1740                 :            :     }
    1741                 :            :   /* FIXME: we could support this.  */
    1742                 :    1071380 :   if (DECL_STRUCT_FUNCTION (current_function_decl)->static_chain_decl)
    1743                 :            :     {
    1744                 :       1180 :       if (dump_file)
    1745                 :          0 :         fprintf (dump_file, "Not splitting: nested function.\n");
    1746                 :       1180 :       return 0;
    1747                 :            :     }
    1748                 :            : 
    1749                 :            :   /* See if it makes sense to try to split.
    1750                 :            :      It makes sense to split if we inline, that is if we have direct calls to
    1751                 :            :      handle or direct calls are possibly going to appear as result of indirect
    1752                 :            :      inlining or LTO.  Also handle -fprofile-generate as LTO to allow non-LTO
    1753                 :            :      training for LTO -fprofile-use build.
    1754                 :            : 
    1755                 :            :      Note that we are not completely conservative about disqualifying functions
    1756                 :            :      called once.  It is possible that the caller is called more then once and
    1757                 :            :      then inlining would still benefit.  */
    1758                 :    1070200 :   if ((!node->callers
    1759                 :            :        /* Local functions called once will be completely inlined most of time.  */
    1760                 :     804497 :        || (!node->callers->next_caller && node->local))
    1761                 :     321830 :       && !node->address_taken
    1762                 :     399375 :       && !node->has_aliases_p ()
    1763                 :    1223110 :       && (!flag_lto || !node->externally_visible))
    1764                 :            :     {
    1765                 :     143256 :       if (dump_file)
    1766                 :         20 :         fprintf (dump_file, "Not splitting: not called directly "
    1767                 :            :                  "or called once.\n");
    1768                 :     143256 :       return 0;
    1769                 :            :     }
    1770                 :            : 
    1771                 :            :   /* FIXME: We can actually split if splitting reduces call overhead.  */
    1772                 :     926945 :   if (!flag_inline_small_functions
    1773                 :     926945 :       && !DECL_DECLARED_INLINE_P (current_function_decl))
    1774                 :            :     {
    1775                 :         55 :       if (dump_file)
    1776                 :          0 :         fprintf (dump_file, "Not splitting: not autoinlining and function"
    1777                 :            :                  " is not inline.\n");
    1778                 :         55 :       return 0;
    1779                 :            :     }
    1780                 :            : 
    1781                 :     926890 :   if (lookup_attribute ("noinline", DECL_ATTRIBUTES (current_function_decl)))
    1782                 :            :     {
    1783                 :          0 :       if (dump_file)
    1784                 :          0 :         fprintf (dump_file, "Not splitting: function is noinline.\n");
    1785                 :          0 :       return 0;
    1786                 :            :     }
    1787                 :     926890 :   if (lookup_attribute ("section", DECL_ATTRIBUTES (current_function_decl)))
    1788                 :            :     {
    1789                 :          7 :       if (dump_file)
    1790                 :          0 :         fprintf (dump_file, "Not splitting: function is in user defined "
    1791                 :            :                  "section.\n");
    1792                 :          7 :       return 0;
    1793                 :            :     }
    1794                 :            : 
    1795                 :            :   /* We enforce splitting after loop headers when profile info is not
    1796                 :            :      available.  */
    1797                 :     926883 :   if (profile_status_for_fn (cfun) != PROFILE_READ)
    1798                 :     926815 :     mark_dfs_back_edges ();
    1799                 :            : 
    1800                 :            :   /* Initialize bitmap to track forbidden calls.  */
    1801                 :     926883 :   forbidden_dominators = BITMAP_ALLOC (NULL);
    1802                 :     926883 :   calculate_dominance_info (CDI_DOMINATORS);
    1803                 :            : 
    1804                 :            :   /* Compute local info about basic blocks and determine function size/time.  */
    1805                 :     926883 :   bb_info_vec.safe_grow_cleared (last_basic_block_for_fn (cfun) + 1);
    1806                 :     926883 :   best_split_point.split_bbs = NULL;
    1807                 :     926883 :   basic_block return_bb = find_return_bb ();
    1808                 :     926883 :   int tsan_exit_found = -1;
    1809                 :    3471900 :   FOR_EACH_BB_FN (bb, cfun)
    1810                 :            :     {
    1811                 :    2545030 :       sreal time = 0;
    1812                 :    2545030 :       int size = 0;
    1813                 :    2545030 :       sreal freq = bb->count.to_sreal_scale
    1814                 :    2545030 :                          (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count);
    1815                 :            : 
    1816                 :    2545030 :       if (dump_file && (dump_flags & TDF_DETAILS))
    1817                 :          6 :         fprintf (dump_file, "Basic block %i\n", bb->index);
    1818                 :            : 
    1819                 :   31323400 :       for (bsi = gsi_start_bb (bb); !gsi_end_p (bsi); gsi_next (&bsi))
    1820                 :            :         {
    1821                 :   26233300 :           sreal this_time;
    1822                 :   26233300 :           int this_size;
    1823                 :   26233300 :           gimple *stmt = gsi_stmt (bsi);
    1824                 :            : 
    1825                 :   26233300 :           this_size = estimate_num_insns (stmt, &eni_size_weights);
    1826                 :   26233300 :           this_time = (sreal)estimate_num_insns (stmt, &eni_time_weights)
    1827                 :   26233300 :                          * freq;
    1828                 :   26233300 :           size += this_size;
    1829                 :   26233300 :           time += this_time;
    1830                 :   26233300 :           check_forbidden_calls (stmt);
    1831                 :            : 
    1832                 :   26233300 :           if (dump_file && (dump_flags & TDF_DETAILS))
    1833                 :            :             {
    1834                 :         13 :               fprintf (dump_file, "  freq:%4.2f size:%3i time:%4.2f ",
    1835                 :            :                        freq.to_double (), this_size, this_time.to_double ());
    1836                 :         13 :               print_gimple_stmt (dump_file, stmt, 0);
    1837                 :            :             }
    1838                 :            : 
    1839                 :   26233300 :           if ((flag_sanitize & SANITIZE_THREAD)
    1840                 :   26233300 :               && gimple_call_internal_p (stmt, IFN_TSAN_FUNC_EXIT))
    1841                 :            :             {
    1842                 :            :               /* We handle TSAN_FUNC_EXIT for splitting either in the
    1843                 :            :                  return_bb, or in its immediate predecessors.  */
    1844                 :         92 :               if ((bb != return_bb && !find_edge (bb, return_bb))
    1845                 :        196 :                   || (tsan_exit_found != -1
    1846                 :          2 :                       && tsan_exit_found != (bb != return_bb)))
    1847                 :            :                 {
    1848                 :         16 :                   if (dump_file)
    1849                 :          0 :                     fprintf (dump_file, "Not splitting: TSAN_FUNC_EXIT"
    1850                 :            :                              " in unexpected basic block.\n");
    1851                 :         16 :                   BITMAP_FREE (forbidden_dominators);
    1852                 :         16 :                   bb_info_vec.release ();
    1853                 :         16 :                   return 0;
    1854                 :            :                 }
    1855                 :        104 :               tsan_exit_found = bb != return_bb;
    1856                 :            :             }
    1857                 :            :         }
    1858                 :    2545010 :       overall_time += time;
    1859                 :    2545010 :       overall_size += size;
    1860                 :    2545010 :       bb_info_vec[bb->index].time = time;
    1861                 :    2545010 :       bb_info_vec[bb->index].size = size;
    1862                 :            :     }
    1863                 :     926867 :   find_split_points (return_bb, overall_time, overall_size);
    1864                 :     926867 :   if (best_split_point.split_bbs)
    1865                 :            :     {
    1866                 :      23014 :       split_function (return_bb, &best_split_point, tsan_exit_found == 1);
    1867                 :      23014 :       BITMAP_FREE (best_split_point.ssa_names_to_pass);
    1868                 :      23014 :       BITMAP_FREE (best_split_point.split_bbs);
    1869                 :      23014 :       todo = TODO_update_ssa | TODO_cleanup_cfg;
    1870                 :            :     }
    1871                 :     926867 :   BITMAP_FREE (forbidden_dominators);
    1872                 :     926867 :   bb_info_vec.release ();
    1873                 :     926867 :   return todo;
    1874                 :            : }
    1875                 :            : 
    1876                 :            : namespace {
    1877                 :            : 
    1878                 :            : const pass_data pass_data_split_functions =
    1879                 :            : {
    1880                 :            :   GIMPLE_PASS, /* type */
    1881                 :            :   "fnsplit", /* name */
    1882                 :            :   OPTGROUP_NONE, /* optinfo_flags */
    1883                 :            :   TV_IPA_FNSPLIT, /* tv_id */
    1884                 :            :   PROP_cfg, /* properties_required */
    1885                 :            :   0, /* properties_provided */
    1886                 :            :   0, /* properties_destroyed */
    1887                 :            :   0, /* todo_flags_start */
    1888                 :            :   0, /* todo_flags_finish */
    1889                 :            : };
    1890                 :            : 
    1891                 :            : class pass_split_functions : public gimple_opt_pass
    1892                 :            : {
    1893                 :            : public:
    1894                 :     200540 :   pass_split_functions (gcc::context *ctxt)
    1895                 :     401080 :     : gimple_opt_pass (pass_data_split_functions, ctxt)
    1896                 :            :   {}
    1897                 :            : 
    1898                 :            :   /* opt_pass methods: */
    1899                 :            :   virtual bool gate (function *);
    1900                 :    1341470 :   virtual unsigned int execute (function *)
    1901                 :            :     {
    1902                 :    1341470 :       return execute_split_functions ();
    1903                 :            :     }
    1904                 :            : 
    1905                 :            : }; // class pass_split_functions
    1906                 :            : 
    1907                 :            : bool
    1908                 :    1584130 : pass_split_functions::gate (function *)
    1909                 :            : {
    1910                 :            :   /* When doing profile feedback, we want to execute the pass after profiling
    1911                 :            :      is read.  So disable one in early optimization.  */
    1912                 :    1584130 :   return (flag_partial_inlining
    1913                 :    1584130 :           && !profile_arc_flag && !flag_branch_probabilities);
    1914                 :            : }
    1915                 :            : 
    1916                 :            : } // anon namespace
    1917                 :            : 
    1918                 :            : gimple_opt_pass *
    1919                 :     200540 : make_pass_split_functions (gcc::context *ctxt)
    1920                 :            : {
    1921                 :     200540 :   return new pass_split_functions (ctxt);
    1922                 :            : }
    1923                 :            : 
    1924                 :            : /* Execute function splitting pass.  */
    1925                 :            : 
    1926                 :            : static unsigned int
    1927                 :        374 : execute_feedback_split_functions (void)
    1928                 :            : {
    1929                 :          0 :   unsigned int retval = execute_split_functions ();
    1930                 :        374 :   if (retval)
    1931                 :          5 :     retval |= TODO_rebuild_cgraph_edges;
    1932                 :        374 :   return retval;
    1933                 :            : }
    1934                 :            : 
    1935                 :            : namespace {
    1936                 :            : 
    1937                 :            : const pass_data pass_data_feedback_split_functions =
    1938                 :            : {
    1939                 :            :   GIMPLE_PASS, /* type */
    1940                 :            :   "feedback_fnsplit", /* name */
    1941                 :            :   OPTGROUP_NONE, /* optinfo_flags */
    1942                 :            :   TV_IPA_FNSPLIT, /* tv_id */
    1943                 :            :   PROP_cfg, /* properties_required */
    1944                 :            :   0, /* properties_provided */
    1945                 :            :   0, /* properties_destroyed */
    1946                 :            :   0, /* todo_flags_start */
    1947                 :            :   0, /* todo_flags_finish */
    1948                 :            : };
    1949                 :            : 
    1950                 :            : class pass_feedback_split_functions : public gimple_opt_pass
    1951                 :            : {
    1952                 :            : public:
    1953                 :     200540 :   pass_feedback_split_functions (gcc::context *ctxt)
    1954                 :     401080 :     : gimple_opt_pass (pass_data_feedback_split_functions, ctxt)
    1955                 :            :   {}
    1956                 :            : 
    1957                 :            :   /* opt_pass methods: */
    1958                 :            :   virtual bool gate (function *);
    1959                 :        374 :   virtual unsigned int execute (function *)
    1960                 :            :     {
    1961                 :        374 :       return execute_feedback_split_functions ();
    1962                 :            :     }
    1963                 :            : 
    1964                 :            : }; // class pass_feedback_split_functions
    1965                 :            : 
    1966                 :            : bool
    1967                 :       2055 : pass_feedback_split_functions::gate (function *)
    1968                 :            : {
    1969                 :            :   /* We don't need to split when profiling at all, we are producing
    1970                 :            :      lousy code anyway.  */
    1971                 :       2055 :   return (flag_partial_inlining
    1972                 :       2055 :           && flag_branch_probabilities);
    1973                 :            : }
    1974                 :            : 
    1975                 :            : } // anon namespace
    1976                 :            : 
    1977                 :            : gimple_opt_pass *
    1978                 :     200540 : make_pass_feedback_split_functions (gcc::context *ctxt)
    1979                 :            : {
    1980                 :     200540 :   return new pass_feedback_split_functions (ctxt);
    1981                 :            : }

Generated by: LCOV version 1.0

LCOV profile is generated on x86_64 machine using following configure options: configure --disable-bootstrap --enable-coverage=opt --enable-languages=c,c++,fortran,go,jit,lto --enable-host-shared. GCC test suite is run with the built compiler.