LCOV - code coverage report
Current view: top level - gcc - tree-sra.c (source / functions) Hit Total Coverage
Test: gcc.info Lines: 1914 2002 95.6 %
Date: 2020-03-28 11:57:23 Functions: 84 95 88.4 %
Legend: Lines: hit not hit | Branches: + taken - not taken # not executed Branches: 0 0 -

           Branch data     Line data    Source code
       1                 :            : /* Scalar Replacement of Aggregates (SRA) converts some structure
       2                 :            :    references into scalar references, exposing them to the scalar
       3                 :            :    optimizers.
       4                 :            :    Copyright (C) 2008-2020 Free Software Foundation, Inc.
       5                 :            :    Contributed by Martin Jambor <mjambor@suse.cz>
       6                 :            : 
       7                 :            : This file is part of GCC.
       8                 :            : 
       9                 :            : GCC is free software; you can redistribute it and/or modify it under
      10                 :            : the terms of the GNU General Public License as published by the Free
      11                 :            : Software Foundation; either version 3, or (at your option) any later
      12                 :            : version.
      13                 :            : 
      14                 :            : GCC is distributed in the hope that it will be useful, but WITHOUT ANY
      15                 :            : WARRANTY; without even the implied warranty of MERCHANTABILITY or
      16                 :            : FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
      17                 :            : for more details.
      18                 :            : 
      19                 :            : You should have received a copy of the GNU General Public License
      20                 :            : along with GCC; see the file COPYING3.  If not see
      21                 :            : <http://www.gnu.org/licenses/>.  */
      22                 :            : 
      23                 :            : /* This file implements Scalar Reduction of Aggregates (SRA).  SRA is run
      24                 :            :    twice, once in the early stages of compilation (early SRA) and once in the
      25                 :            :    late stages (late SRA).  The aim of both is to turn references to scalar
      26                 :            :    parts of aggregates into uses of independent scalar variables.
      27                 :            : 
      28                 :            :    The two passes are nearly identical, the only difference is that early SRA
      29                 :            :    does not scalarize unions which are used as the result in a GIMPLE_RETURN
      30                 :            :    statement because together with inlining this can lead to weird type
      31                 :            :    conversions.
      32                 :            : 
      33                 :            :    Both passes operate in four stages:
      34                 :            : 
      35                 :            :    1. The declarations that have properties which make them candidates for
      36                 :            :       scalarization are identified in function find_var_candidates().  The
      37                 :            :       candidates are stored in candidate_bitmap.
      38                 :            : 
      39                 :            :    2. The function body is scanned.  In the process, declarations which are
      40                 :            :       used in a manner that prevent their scalarization are removed from the
      41                 :            :       candidate bitmap.  More importantly, for every access into an aggregate,
      42                 :            :       an access structure (struct access) is created by create_access() and
      43                 :            :       stored in a vector associated with the aggregate.  Among other
      44                 :            :       information, the aggregate declaration, the offset and size of the access
      45                 :            :       and its type are stored in the structure.
      46                 :            : 
      47                 :            :       On a related note, assign_link structures are created for every assign
      48                 :            :       statement between candidate aggregates and attached to the related
      49                 :            :       accesses.
      50                 :            : 
      51                 :            :    3. The vectors of accesses are analyzed.  They are first sorted according to
      52                 :            :       their offset and size and then scanned for partially overlapping accesses
      53                 :            :       (i.e. those which overlap but one is not entirely within another).  Such
      54                 :            :       an access disqualifies the whole aggregate from being scalarized.
      55                 :            : 
      56                 :            :       If there is no such inhibiting overlap, a representative access structure
      57                 :            :       is chosen for every unique combination of offset and size.  Afterwards,
      58                 :            :       the pass builds a set of trees from these structures, in which children
      59                 :            :       of an access are within their parent (in terms of offset and size).
      60                 :            : 
      61                 :            :       Then accesses  are propagated  whenever possible (i.e.  in cases  when it
      62                 :            :       does not create a partially overlapping access) across assign_links from
      63                 :            :       the right hand side to the left hand side.
      64                 :            : 
      65                 :            :       Then the set of trees for each declaration is traversed again and those
      66                 :            :       accesses which should be replaced by a scalar are identified.
      67                 :            : 
      68                 :            :    4. The function is traversed again, and for every reference into an
      69                 :            :       aggregate that has some component which is about to be scalarized,
      70                 :            :       statements are amended and new statements are created as necessary.
      71                 :            :       Finally, if a parameter got scalarized, the scalar replacements are
      72                 :            :       initialized with values from respective parameter aggregates.  */
      73                 :            : 
      74                 :            : #include "config.h"
      75                 :            : #include "system.h"
      76                 :            : #include "coretypes.h"
      77                 :            : #include "backend.h"
      78                 :            : #include "target.h"
      79                 :            : #include "rtl.h"
      80                 :            : #include "tree.h"
      81                 :            : #include "gimple.h"
      82                 :            : #include "predict.h"
      83                 :            : #include "alloc-pool.h"
      84                 :            : #include "tree-pass.h"
      85                 :            : #include "ssa.h"
      86                 :            : #include "cgraph.h"
      87                 :            : #include "gimple-pretty-print.h"
      88                 :            : #include "alias.h"
      89                 :            : #include "fold-const.h"
      90                 :            : #include "tree-eh.h"
      91                 :            : #include "stor-layout.h"
      92                 :            : #include "gimplify.h"
      93                 :            : #include "gimple-iterator.h"
      94                 :            : #include "gimplify-me.h"
      95                 :            : #include "gimple-walk.h"
      96                 :            : #include "tree-cfg.h"
      97                 :            : #include "tree-dfa.h"
      98                 :            : #include "tree-ssa.h"
      99                 :            : #include "dbgcnt.h"
     100                 :            : #include "builtins.h"
     101                 :            : #include "tree-sra.h"
     102                 :            : 
     103                 :            : 
     104                 :            : /* Enumeration of all aggregate reductions we can do.  */
     105                 :            : enum sra_mode { SRA_MODE_EARLY_IPA,   /* early call regularization */
     106                 :            :                 SRA_MODE_EARLY_INTRA, /* early intraprocedural SRA */
     107                 :            :                 SRA_MODE_INTRA };     /* late intraprocedural SRA */
     108                 :            : 
     109                 :            : /* Global variable describing which aggregate reduction we are performing at
     110                 :            :    the moment.  */
     111                 :            : static enum sra_mode sra_mode;
     112                 :            : 
     113                 :            : struct assign_link;
     114                 :            : 
     115                 :            : /* ACCESS represents each access to an aggregate variable (as a whole or a
     116                 :            :    part).  It can also represent a group of accesses that refer to exactly the
     117                 :            :    same fragment of an aggregate (i.e. those that have exactly the same offset
     118                 :            :    and size).  Such representatives for a single aggregate, once determined,
     119                 :            :    are linked in a linked list and have the group fields set.
     120                 :            : 
     121                 :            :    Moreover, when doing intraprocedural SRA, a tree is built from those
     122                 :            :    representatives (by the means of first_child and next_sibling pointers), in
     123                 :            :    which all items in a subtree are "within" the root, i.e. their offset is
     124                 :            :    greater or equal to offset of the root and offset+size is smaller or equal
     125                 :            :    to offset+size of the root.  Children of an access are sorted by offset.
     126                 :            : 
     127                 :            :    Note that accesses to parts of vector and complex number types always
     128                 :            :    represented by an access to the whole complex number or a vector.  It is a
     129                 :            :    duty of the modifying functions to replace them appropriately.  */
     130                 :            : 
     131                 :            : struct access
     132                 :            : {
     133                 :            :   /* Values returned by  `get_ref_base_and_extent' for each component reference
     134                 :            :      If EXPR isn't a component reference  just set `BASE = EXPR', `OFFSET = 0',
     135                 :            :      `SIZE = TREE_SIZE (TREE_TYPE (expr))'.  */
     136                 :            :   HOST_WIDE_INT offset;
     137                 :            :   HOST_WIDE_INT size;
     138                 :            :   tree base;
     139                 :            : 
     140                 :            :   /* Expression.  It is context dependent so do not use it to create new
     141                 :            :      expressions to access the original aggregate.  See PR 42154 for a
     142                 :            :      testcase.  */
     143                 :            :   tree expr;
     144                 :            :   /* Type.  */
     145                 :            :   tree type;
     146                 :            : 
     147                 :            :   /* The statement this access belongs to.  */
     148                 :            :   gimple *stmt;
     149                 :            : 
     150                 :            :   /* Next group representative for this aggregate. */
     151                 :            :   struct access *next_grp;
     152                 :            : 
     153                 :            :   /* Pointer to the group representative.  Pointer to itself if the struct is
     154                 :            :      the representative.  */
     155                 :            :   struct access *group_representative;
     156                 :            : 
     157                 :            :   /* After access tree has been constructed, this points to the parent of the
     158                 :            :      current access, if there is one.  NULL for roots.  */
     159                 :            :   struct access *parent;
     160                 :            : 
     161                 :            :   /* If this access has any children (in terms of the definition above), this
     162                 :            :      points to the first one.  */
     163                 :            :   struct access *first_child;
     164                 :            : 
     165                 :            :   /* In intraprocedural SRA, pointer to the next sibling in the access tree as
     166                 :            :      described above.  */
     167                 :            :   struct access *next_sibling;
     168                 :            : 
     169                 :            :   /* Pointers to the first and last element in the linked list of assign
     170                 :            :      links for propagation from LHS to RHS.  */
     171                 :            :   struct assign_link *first_rhs_link, *last_rhs_link;
     172                 :            : 
     173                 :            :   /* Pointers to the first and last element in the linked list of assign
     174                 :            :      links for propagation from LHS to RHS.  */
     175                 :            :   struct assign_link *first_lhs_link, *last_lhs_link;
     176                 :            : 
     177                 :            :   /* Pointer to the next access in the work queues.  */
     178                 :            :   struct access *next_rhs_queued, *next_lhs_queued;
     179                 :            : 
     180                 :            :   /* Replacement variable for this access "region."  Never to be accessed
     181                 :            :      directly, always only by the means of get_access_replacement() and only
     182                 :            :      when grp_to_be_replaced flag is set.  */
     183                 :            :   tree replacement_decl;
     184                 :            : 
     185                 :            :   /* Is this access made in reverse storage order? */
     186                 :            :   unsigned reverse : 1;
     187                 :            : 
     188                 :            :   /* Is this particular access write access? */
     189                 :            :   unsigned write : 1;
     190                 :            : 
     191                 :            :   /* Is this access currently in the rhs work queue?  */
     192                 :            :   unsigned grp_rhs_queued : 1;
     193                 :            : 
     194                 :            :   /* Is this access currently in the lhs work queue?  */
     195                 :            :   unsigned grp_lhs_queued : 1;
     196                 :            : 
     197                 :            :   /* Does this group contain a write access?  This flag is propagated down the
     198                 :            :      access tree.  */
     199                 :            :   unsigned grp_write : 1;
     200                 :            : 
     201                 :            :   /* Does this group contain a read access?  This flag is propagated down the
     202                 :            :      access tree.  */
     203                 :            :   unsigned grp_read : 1;
     204                 :            : 
     205                 :            :   /* Does this group contain a read access that comes from an assignment
     206                 :            :      statement?  This flag is propagated down the access tree.  */
     207                 :            :   unsigned grp_assignment_read : 1;
     208                 :            : 
     209                 :            :   /* Does this group contain a write access that comes from an assignment
     210                 :            :      statement?  This flag is propagated down the access tree.  */
     211                 :            :   unsigned grp_assignment_write : 1;
     212                 :            : 
     213                 :            :   /* Does this group contain a read access through a scalar type?  This flag is
     214                 :            :      not propagated in the access tree in any direction.  */
     215                 :            :   unsigned grp_scalar_read : 1;
     216                 :            : 
     217                 :            :   /* Does this group contain a write access through a scalar type?  This flag
     218                 :            :      is not propagated in the access tree in any direction.  */
     219                 :            :   unsigned grp_scalar_write : 1;
     220                 :            : 
     221                 :            :   /* In a root of an access tree, true means that the entire tree should be
     222                 :            :      totally scalarized - that all scalar leafs should be scalarized and
     223                 :            :      non-root grp_total_scalarization accesses should be honored.  Otherwise,
     224                 :            :      non-root accesses with grp_total_scalarization should never get scalar
     225                 :            :      replacements.  */
     226                 :            :   unsigned grp_total_scalarization : 1;
     227                 :            : 
     228                 :            :   /* Other passes of the analysis use this bit to make function
     229                 :            :      analyze_access_subtree create scalar replacements for this group if
     230                 :            :      possible.  */
     231                 :            :   unsigned grp_hint : 1;
     232                 :            : 
     233                 :            :   /* Is the subtree rooted in this access fully covered by scalar
     234                 :            :      replacements?  */
     235                 :            :   unsigned grp_covered : 1;
     236                 :            : 
     237                 :            :   /* If set to true, this access and all below it in an access tree must not be
     238                 :            :      scalarized.  */
     239                 :            :   unsigned grp_unscalarizable_region : 1;
     240                 :            : 
     241                 :            :   /* Whether data have been written to parts of the aggregate covered by this
     242                 :            :      access which is not to be scalarized.  This flag is propagated up in the
     243                 :            :      access tree.  */
     244                 :            :   unsigned grp_unscalarized_data : 1;
     245                 :            : 
     246                 :            :   /* Set if all accesses in the group consist of the same chain of
     247                 :            :      COMPONENT_REFs and ARRAY_REFs.  */
     248                 :            :   unsigned grp_same_access_path : 1;
     249                 :            : 
     250                 :            :   /* Does this access and/or group contain a write access through a
     251                 :            :      BIT_FIELD_REF?  */
     252                 :            :   unsigned grp_partial_lhs : 1;
     253                 :            : 
     254                 :            :   /* Set when a scalar replacement should be created for this variable.  */
     255                 :            :   unsigned grp_to_be_replaced : 1;
     256                 :            : 
     257                 :            :   /* Set when we want a replacement for the sole purpose of having it in
     258                 :            :      generated debug statements.  */
     259                 :            :   unsigned grp_to_be_debug_replaced : 1;
     260                 :            : 
     261                 :            :   /* Should TREE_NO_WARNING of a replacement be set?  */
     262                 :            :   unsigned grp_no_warning : 1;
     263                 :            : };
     264                 :            : 
     265                 :            : typedef struct access *access_p;
     266                 :            : 
     267                 :            : 
     268                 :            : /* Alloc pool for allocating access structures.  */
     269                 :            : static object_allocator<struct access> access_pool ("SRA accesses");
     270                 :            : 
     271                 :            : /* A structure linking lhs and rhs accesses from an aggregate assignment.  They
     272                 :            :    are used to propagate subaccesses from rhs to lhs and vice versa as long as
     273                 :            :    they don't conflict with what is already there.  In the RHS->LHS direction,
     274                 :            :    we also propagate grp_write flag to lazily mark that the access contains any
     275                 :            :    meaningful data.  */
     276                 :            : struct assign_link
     277                 :            : {
     278                 :            :   struct access *lacc, *racc;
     279                 :            :   struct assign_link *next_rhs, *next_lhs;
     280                 :            : };
     281                 :            : 
     282                 :            : /* Alloc pool for allocating assign link structures.  */
     283                 :            : static object_allocator<assign_link> assign_link_pool ("SRA links");
     284                 :            : 
     285                 :            : /* Base (tree) -> Vector (vec<access_p> *) map.  */
     286                 :            : static hash_map<tree, auto_vec<access_p> > *base_access_vec;
     287                 :            : 
     288                 :            : /* Hash to limit creation of artificial accesses */
     289                 :            : static hash_map<tree, unsigned> *propagation_budget;
     290                 :            : 
     291                 :            : /* Candidate hash table helpers.  */
     292                 :            : 
     293                 :            : struct uid_decl_hasher : nofree_ptr_hash <tree_node>
     294                 :            : {
     295                 :            :   static inline hashval_t hash (const tree_node *);
     296                 :            :   static inline bool equal (const tree_node *, const tree_node *);
     297                 :            : };
     298                 :            : 
     299                 :            : /* Hash a tree in a uid_decl_map.  */
     300                 :            : 
     301                 :            : inline hashval_t
     302                 :   23736000 : uid_decl_hasher::hash (const tree_node *item)
     303                 :            : {
     304                 :   23736000 :   return item->decl_minimal.uid;
     305                 :            : }
     306                 :            : 
     307                 :            : /* Return true if the DECL_UID in both trees are equal.  */
     308                 :            : 
     309                 :            : inline bool
     310                 :   37845300 : uid_decl_hasher::equal (const tree_node *a, const tree_node *b)
     311                 :            : {
     312                 :   37845300 :   return (a->decl_minimal.uid == b->decl_minimal.uid);
     313                 :            : }
     314                 :            : 
     315                 :            : /* Set of candidates.  */
     316                 :            : static bitmap candidate_bitmap;
     317                 :            : static hash_table<uid_decl_hasher> *candidates;
     318                 :            : 
     319                 :            : /* For a candidate UID return the candidates decl.  */
     320                 :            : 
     321                 :            : static inline tree
     322                 :   11065300 : candidate (unsigned uid)
     323                 :            : {
     324                 :   11065300 :  tree_node t;
     325                 :   11065300 :  t.decl_minimal.uid = uid;
     326                 :   22130600 :  return candidates->find_with_hash (&t, static_cast <hashval_t> (uid));
     327                 :            : }
     328                 :            : 
     329                 :            : /* Bitmap of candidates which we should try to entirely scalarize away and
     330                 :            :    those which cannot be (because they are and need be used as a whole).  */
     331                 :            : static bitmap should_scalarize_away_bitmap, cannot_scalarize_away_bitmap;
     332                 :            : 
     333                 :            : /* Bitmap of candidates in the constant pool, which cannot be scalarized
     334                 :            :    because this would produce non-constant expressions (e.g. Ada).  */
     335                 :            : static bitmap disqualified_constants;
     336                 :            : 
     337                 :            : /* Obstack for creation of fancy names.  */
     338                 :            : static struct obstack name_obstack;
     339                 :            : 
     340                 :            : /* Head of a linked list of accesses that need to have its subaccesses
     341                 :            :    propagated to their assignment counterparts. */
     342                 :            : static struct access *rhs_work_queue_head, *lhs_work_queue_head;
     343                 :            : 
     344                 :            : /* Dump contents of ACCESS to file F in a human friendly way.  If GRP is true,
     345                 :            :    representative fields are dumped, otherwise those which only describe the
     346                 :            :    individual access are.  */
     347                 :            : 
     348                 :            : static struct
     349                 :            : {
     350                 :            :   /* Number of processed aggregates is readily available in
     351                 :            :      analyze_all_variable_accesses and so is not stored here.  */
     352                 :            : 
     353                 :            :   /* Number of created scalar replacements.  */
     354                 :            :   int replacements;
     355                 :            : 
     356                 :            :   /* Number of times sra_modify_expr or sra_modify_assign themselves changed an
     357                 :            :      expression.  */
     358                 :            :   int exprs;
     359                 :            : 
     360                 :            :   /* Number of statements created by generate_subtree_copies.  */
     361                 :            :   int subtree_copies;
     362                 :            : 
     363                 :            :   /* Number of statements created by load_assign_lhs_subreplacements.  */
     364                 :            :   int subreplacements;
     365                 :            : 
     366                 :            :   /* Number of times sra_modify_assign has deleted a statement.  */
     367                 :            :   int deleted;
     368                 :            : 
     369                 :            :   /* Number of times sra_modify_assign has to deal with subaccesses of LHS and
     370                 :            :      RHS reparately due to type conversions or nonexistent matching
     371                 :            :      references.  */
     372                 :            :   int separate_lhs_rhs_handling;
     373                 :            : 
     374                 :            :   /* Number of parameters that were removed because they were unused.  */
     375                 :            :   int deleted_unused_parameters;
     376                 :            : 
     377                 :            :   /* Number of scalars passed as parameters by reference that have been
     378                 :            :      converted to be passed by value.  */
     379                 :            :   int scalar_by_ref_to_by_val;
     380                 :            : 
     381                 :            :   /* Number of aggregate parameters that were replaced by one or more of their
     382                 :            :      components.  */
     383                 :            :   int aggregate_params_reduced;
     384                 :            : 
     385                 :            :   /* Numbber of components created when splitting aggregate parameters.  */
     386                 :            :   int param_reductions_created;
     387                 :            : } sra_stats;
     388                 :            : 
     389                 :            : static void
     390                 :         36 : dump_access (FILE *f, struct access *access, bool grp)
     391                 :            : {
     392                 :         36 :   fprintf (f, "access { ");
     393                 :         36 :   fprintf (f, "base = (%d)'", DECL_UID (access->base));
     394                 :         36 :   print_generic_expr (f, access->base);
     395                 :         36 :   fprintf (f, "', offset = " HOST_WIDE_INT_PRINT_DEC, access->offset);
     396                 :         36 :   fprintf (f, ", size = " HOST_WIDE_INT_PRINT_DEC, access->size);
     397                 :         36 :   fprintf (f, ", expr = ");
     398                 :         36 :   print_generic_expr (f, access->expr);
     399                 :         36 :   fprintf (f, ", type = ");
     400                 :         36 :   print_generic_expr (f, access->type);
     401                 :         36 :   fprintf (f, ", reverse = %d", access->reverse);
     402                 :         36 :   if (grp)
     403                 :         36 :     fprintf (f, ", grp_read = %d, grp_write = %d, grp_assignment_read = %d, "
     404                 :            :              "grp_assignment_write = %d, grp_scalar_read = %d, "
     405                 :            :              "grp_scalar_write = %d, grp_total_scalarization = %d, "
     406                 :            :              "grp_hint = %d, grp_covered = %d, "
     407                 :            :              "grp_unscalarizable_region = %d, grp_unscalarized_data = %d, "
     408                 :            :              "grp_same_access_path = %d, grp_partial_lhs = %d, "
     409                 :            :              "grp_to_be_replaced = %d, grp_to_be_debug_replaced = %d}\n",
     410                 :         36 :              access->grp_read, access->grp_write, access->grp_assignment_read,
     411                 :         36 :              access->grp_assignment_write, access->grp_scalar_read,
     412                 :         36 :              access->grp_scalar_write, access->grp_total_scalarization,
     413                 :         36 :              access->grp_hint, access->grp_covered,
     414                 :         36 :              access->grp_unscalarizable_region, access->grp_unscalarized_data,
     415                 :         36 :              access->grp_same_access_path, access->grp_partial_lhs,
     416                 :         36 :              access->grp_to_be_replaced, access->grp_to_be_debug_replaced);
     417                 :            :   else
     418                 :          0 :     fprintf (f, ", write = %d, grp_total_scalarization = %d, "
     419                 :            :              "grp_partial_lhs = %d}\n",
     420                 :          0 :              access->write, access->grp_total_scalarization,
     421                 :          0 :              access->grp_partial_lhs);
     422                 :         36 : }
     423                 :            : 
     424                 :            : /* Dump a subtree rooted in ACCESS to file F, indent by LEVEL.  */
     425                 :            : 
     426                 :            : static void
     427                 :         18 : dump_access_tree_1 (FILE *f, struct access *access, int level)
     428                 :            : {
     429                 :         36 :   do
     430                 :            :     {
     431                 :         36 :       int i;
     432                 :            : 
     433                 :         62 :       for (i = 0; i < level; i++)
     434                 :         26 :         fputs ("* ", f);
     435                 :            : 
     436                 :         36 :       dump_access (f, access, true);
     437                 :            : 
     438                 :         36 :       if (access->first_child)
     439                 :          8 :         dump_access_tree_1 (f, access->first_child, level + 1);
     440                 :            : 
     441                 :         36 :       access = access->next_sibling;
     442                 :            :     }
     443                 :         36 :   while (access);
     444                 :         18 : }
     445                 :            : 
     446                 :            : /* Dump all access trees for a variable, given the pointer to the first root in
     447                 :            :    ACCESS.  */
     448                 :            : 
     449                 :            : static void
     450                 :          0 : dump_access_tree (FILE *f, struct access *access)
     451                 :            : {
     452                 :         19 :   for (; access; access = access->next_grp)
     453                 :         10 :     dump_access_tree_1 (f, access, 0);
     454                 :          0 : }
     455                 :            : 
     456                 :            : /* Return true iff ACC is non-NULL and has subaccesses.  */
     457                 :            : 
     458                 :            : static inline bool
     459                 :   10461000 : access_has_children_p (struct access *acc)
     460                 :            : {
     461                 :    5748920 :   return acc && acc->first_child;
     462                 :            : }
     463                 :            : 
     464                 :            : /* Return true iff ACC is (partly) covered by at least one replacement.  */
     465                 :            : 
     466                 :            : static bool
     467                 :        500 : access_has_replacements_p (struct access *acc)
     468                 :            : {
     469                 :        500 :   struct access *child;
     470                 :        500 :   if (acc->grp_to_be_replaced)
     471                 :            :     return true;
     472                 :        475 :   for (child = acc->first_child; child; child = child->next_sibling)
     473                 :         37 :     if (access_has_replacements_p (child))
     474                 :            :       return true;
     475                 :            :   return false;
     476                 :            : }
     477                 :            : 
     478                 :            : /* Return a vector of pointers to accesses for the variable given in BASE or
     479                 :            :    NULL if there is none.  */
     480                 :            : 
     481                 :            : static vec<access_p> *
     482                 :   17656100 : get_base_access_vector (tree base)
     483                 :            : {
     484                 :          0 :   return base_access_vec->get (base);
     485                 :            : }
     486                 :            : 
     487                 :            : /* Find an access with required OFFSET and SIZE in a subtree of accesses rooted
     488                 :            :    in ACCESS.  Return NULL if it cannot be found.  */
     489                 :            : 
     490                 :            : static struct access *
     491                 :    7414630 : find_access_in_subtree (struct access *access, HOST_WIDE_INT offset,
     492                 :            :                         HOST_WIDE_INT size)
     493                 :            : {
     494                 :   11474900 :   while (access && (access->offset != offset || access->size != size))
     495                 :            :     {
     496                 :    4060320 :       struct access *child = access->first_child;
     497                 :            : 
     498                 :    7371370 :       while (child && (child->offset + child->size <= offset))
     499                 :    3311050 :         child = child->next_sibling;
     500                 :            :       access = child;
     501                 :            :     }
     502                 :            : 
     503                 :            :   /* Total scalarization does not replace single field structures with their
     504                 :            :      single field but rather creates an access for them underneath.  Look for
     505                 :            :      it.  */
     506                 :    7414630 :   if (access)
     507                 :    7465400 :     while (access->first_child
     508                 :    2550800 :            && access->first_child->offset == offset
     509                 :    9973700 :            && access->first_child->size == size)
     510                 :            :       access = access->first_child;
     511                 :            : 
     512                 :    7414630 :   return access;
     513                 :            : }
     514                 :            : 
     515                 :            : /* Return the first group representative for DECL or NULL if none exists.  */
     516                 :            : 
     517                 :            : static struct access *
     518                 :   14633100 : get_first_repr_for_decl (tree base)
     519                 :            : {
     520                 :   14633100 :   vec<access_p> *access_vec;
     521                 :            : 
     522                 :   14633100 :   access_vec = get_base_access_vector (base);
     523                 :   14633100 :   if (!access_vec)
     524                 :          0 :     return NULL;
     525                 :            : 
     526                 :   14633100 :   return (*access_vec)[0];
     527                 :            : }
     528                 :            : 
     529                 :            : /* Find an access representative for the variable BASE and given OFFSET and
     530                 :            :    SIZE.  Requires that access trees have already been built.  Return NULL if
     531                 :            :    it cannot be found.  */
     532                 :            : 
     533                 :            : static struct access *
     534                 :    6237690 : get_var_base_offset_size_access (tree base, HOST_WIDE_INT offset,
     535                 :            :                                  HOST_WIDE_INT size)
     536                 :            : {
     537                 :    6237690 :   struct access *access;
     538                 :            : 
     539                 :    6237690 :   access = get_first_repr_for_decl (base);
     540                 :    7609550 :   while (access && (access->offset + access->size <= offset))
     541                 :    1371870 :     access = access->next_grp;
     542                 :    6237690 :   if (!access)
     543                 :            :     return NULL;
     544                 :            : 
     545                 :    6237650 :   return find_access_in_subtree (access, offset, size);
     546                 :            : }
     547                 :            : 
     548                 :            : /* Add LINK to the linked list of assign links of RACC.  */
     549                 :            : 
     550                 :            : static void
     551                 :     909118 : add_link_to_rhs (struct access *racc, struct assign_link *link)
     552                 :            : {
     553                 :     909118 :   gcc_assert (link->racc == racc);
     554                 :            : 
     555                 :     909118 :   if (!racc->first_rhs_link)
     556                 :            :     {
     557                 :     909118 :       gcc_assert (!racc->last_rhs_link);
     558                 :     909118 :       racc->first_rhs_link = link;
     559                 :            :     }
     560                 :            :   else
     561                 :          0 :     racc->last_rhs_link->next_rhs = link;
     562                 :            : 
     563                 :     909118 :   racc->last_rhs_link = link;
     564                 :     909118 :   link->next_rhs = NULL;
     565                 :     909118 : }
     566                 :            : 
     567                 :            : /* Add LINK to the linked list of lhs assign links of LACC.  */
     568                 :            : 
     569                 :            : static void
     570                 :     909118 : add_link_to_lhs (struct access *lacc, struct assign_link *link)
     571                 :            : {
     572                 :     909118 :   gcc_assert (link->lacc == lacc);
     573                 :            : 
     574                 :     909118 :   if (!lacc->first_lhs_link)
     575                 :            :     {
     576                 :     909118 :       gcc_assert (!lacc->last_lhs_link);
     577                 :     909118 :       lacc->first_lhs_link = link;
     578                 :            :     }
     579                 :            :   else
     580                 :          0 :     lacc->last_lhs_link->next_lhs = link;
     581                 :            : 
     582                 :     909118 :   lacc->last_lhs_link = link;
     583                 :     909118 :   link->next_lhs = NULL;
     584                 :     909118 : }
     585                 :            : 
     586                 :            : /* Move all link structures in their linked list in OLD_ACC to the linked list
     587                 :            :    in NEW_ACC.  */
     588                 :            : static void
     589                 :    2917120 : relink_to_new_repr (struct access *new_acc, struct access *old_acc)
     590                 :            : {
     591                 :    2917120 :   if (old_acc->first_rhs_link)
     592                 :            :     {
     593                 :            : 
     594                 :     596395 :       if (new_acc->first_rhs_link)
     595                 :            :         {
     596                 :     110398 :           gcc_assert (!new_acc->last_rhs_link->next_rhs);
     597                 :     110398 :           gcc_assert (!old_acc->last_rhs_link
     598                 :            :                       || !old_acc->last_rhs_link->next_rhs);
     599                 :            : 
     600                 :     110398 :           new_acc->last_rhs_link->next_rhs = old_acc->first_rhs_link;
     601                 :     110398 :           new_acc->last_rhs_link = old_acc->last_rhs_link;
     602                 :            :         }
     603                 :            :       else
     604                 :            :         {
     605                 :     485997 :           gcc_assert (!new_acc->last_rhs_link);
     606                 :            : 
     607                 :     485997 :           new_acc->first_rhs_link = old_acc->first_rhs_link;
     608                 :     485997 :           new_acc->last_rhs_link = old_acc->last_rhs_link;
     609                 :            :         }
     610                 :     596395 :       old_acc->first_rhs_link = old_acc->last_rhs_link = NULL;
     611                 :            :     }
     612                 :            :   else
     613                 :    2320720 :     gcc_assert (!old_acc->last_rhs_link);
     614                 :            : 
     615                 :    2917120 :   if (old_acc->first_lhs_link)
     616                 :            :     {
     617                 :            : 
     618                 :     241374 :       if (new_acc->first_lhs_link)
     619                 :            :         {
     620                 :     105188 :           gcc_assert (!new_acc->last_lhs_link->next_lhs);
     621                 :     105188 :           gcc_assert (!old_acc->last_lhs_link
     622                 :            :                       || !old_acc->last_lhs_link->next_lhs);
     623                 :            : 
     624                 :     105188 :           new_acc->last_lhs_link->next_lhs = old_acc->first_lhs_link;
     625                 :     105188 :           new_acc->last_lhs_link = old_acc->last_lhs_link;
     626                 :            :         }
     627                 :            :       else
     628                 :            :         {
     629                 :     136186 :           gcc_assert (!new_acc->last_lhs_link);
     630                 :            : 
     631                 :     136186 :           new_acc->first_lhs_link = old_acc->first_lhs_link;
     632                 :     136186 :           new_acc->last_lhs_link = old_acc->last_lhs_link;
     633                 :            :         }
     634                 :     241374 :       old_acc->first_lhs_link = old_acc->last_lhs_link = NULL;
     635                 :            :     }
     636                 :            :   else
     637                 :    2675740 :     gcc_assert (!old_acc->last_lhs_link);
     638                 :            : 
     639                 :    2917120 : }
     640                 :            : 
     641                 :            : /* Add ACCESS to the work to queue for propagation of subaccesses from RHS to
     642                 :            :    LHS (which is actually a stack).  */
     643                 :            : 
     644                 :            : static void
     645                 :    3063910 : add_access_to_rhs_work_queue (struct access *access)
     646                 :            : {
     647                 :    3063910 :   if (access->first_rhs_link && !access->grp_rhs_queued)
     648                 :            :     {
     649                 :    1162100 :       gcc_assert (!access->next_rhs_queued);
     650                 :    1162100 :       access->next_rhs_queued = rhs_work_queue_head;
     651                 :    1162100 :       access->grp_rhs_queued = 1;
     652                 :    1162100 :       rhs_work_queue_head = access;
     653                 :            :     }
     654                 :    3063910 : }
     655                 :            : 
     656                 :            : /* Add ACCESS to the work to queue for propagation of subaccesses from LHS to
     657                 :            :    RHS (which is actually a stack).  */
     658                 :            : 
     659                 :            : static void
     660                 :    1097010 : add_access_to_lhs_work_queue (struct access *access)
     661                 :            : {
     662                 :    1097010 :   if (access->first_lhs_link && !access->grp_lhs_queued)
     663                 :            :     {
     664                 :     911049 :       gcc_assert (!access->next_lhs_queued);
     665                 :     911049 :       access->next_lhs_queued = lhs_work_queue_head;
     666                 :     911049 :       access->grp_lhs_queued = 1;
     667                 :     911049 :       lhs_work_queue_head = access;
     668                 :            :     }
     669                 :    1097010 : }
     670                 :            : 
     671                 :            : /* Pop an access from the work queue for propagating from RHS to LHS, and
     672                 :            :    return it, assuming there is one.  */
     673                 :            : 
     674                 :            : static struct access *
     675                 :    1162100 : pop_access_from_rhs_work_queue (void)
     676                 :            : {
     677                 :    1162100 :   struct access *access = rhs_work_queue_head;
     678                 :            : 
     679                 :    1162100 :   rhs_work_queue_head = access->next_rhs_queued;
     680                 :    1162100 :   access->next_rhs_queued = NULL;
     681                 :    1162100 :   access->grp_rhs_queued = 0;
     682                 :    1162100 :   return access;
     683                 :            : }
     684                 :            : 
     685                 :            : /* Pop an access from the work queue for propagating from LHS to RHS, and
     686                 :            :    return it, assuming there is one.  */
     687                 :            : 
     688                 :            : static struct access *
     689                 :     911049 : pop_access_from_lhs_work_queue (void)
     690                 :            : {
     691                 :     911049 :   struct access *access = lhs_work_queue_head;
     692                 :            : 
     693                 :     911049 :   lhs_work_queue_head = access->next_lhs_queued;
     694                 :     911049 :   access->next_lhs_queued = NULL;
     695                 :     911049 :   access->grp_lhs_queued = 0;
     696                 :     911049 :   return access;
     697                 :            : }
     698                 :            : 
     699                 :            : /* Allocate necessary structures.  */
     700                 :            : 
     701                 :            : static void
     702                 :    2266800 : sra_initialize (void)
     703                 :            : {
     704                 :    2266800 :   candidate_bitmap = BITMAP_ALLOC (NULL);
     705                 :    4533590 :   candidates = new hash_table<uid_decl_hasher>
     706                 :    4231030 :     (vec_safe_length (cfun->local_decls) / 2);
     707                 :    2266800 :   should_scalarize_away_bitmap = BITMAP_ALLOC (NULL);
     708                 :    2266800 :   cannot_scalarize_away_bitmap = BITMAP_ALLOC (NULL);
     709                 :    2266800 :   disqualified_constants = BITMAP_ALLOC (NULL);
     710                 :    2266800 :   gcc_obstack_init (&name_obstack);
     711                 :    2266800 :   base_access_vec = new hash_map<tree, auto_vec<access_p> >;
     712                 :    2266800 :   memset (&sra_stats, 0, sizeof (sra_stats));
     713                 :    2266800 : }
     714                 :            : 
     715                 :            : /* Deallocate all general structures.  */
     716                 :            : 
     717                 :            : static void
     718                 :    2266800 : sra_deinitialize (void)
     719                 :            : {
     720                 :    2266800 :   BITMAP_FREE (candidate_bitmap);
     721                 :    2266800 :   delete candidates;
     722                 :    2266800 :   candidates = NULL;
     723                 :    2266800 :   BITMAP_FREE (should_scalarize_away_bitmap);
     724                 :    2266800 :   BITMAP_FREE (cannot_scalarize_away_bitmap);
     725                 :    2266800 :   BITMAP_FREE (disqualified_constants);
     726                 :    2266800 :   access_pool.release ();
     727                 :    2266800 :   assign_link_pool.release ();
     728                 :    2266800 :   obstack_free (&name_obstack, NULL);
     729                 :            : 
     730                 :    4533590 :   delete base_access_vec;
     731                 :    2266800 : }
     732                 :            : 
     733                 :            : /* Return true if DECL is a VAR_DECL in the constant pool, false otherwise.  */
     734                 :            : 
     735                 :   24448100 : static bool constant_decl_p (tree decl)
     736                 :            : {
     737                 :   20842100 :   return VAR_P (decl) && DECL_IN_CONSTANT_POOL (decl);
     738                 :            : }
     739                 :            : 
     740                 :            : /* Remove DECL from candidates for SRA and write REASON to the dump file if
     741                 :            :    there is one.  */
     742                 :            : 
     743                 :            : static void
     744                 :    1595020 : disqualify_candidate (tree decl, const char *reason)
     745                 :            : {
     746                 :    1595020 :   if (bitmap_clear_bit (candidate_bitmap, DECL_UID (decl)))
     747                 :    1584820 :     candidates->remove_elt_with_hash (decl, DECL_UID (decl));
     748                 :    1595020 :   if (constant_decl_p (decl))
     749                 :       2000 :     bitmap_set_bit (disqualified_constants, DECL_UID (decl));
     750                 :            : 
     751                 :    1595020 :   if (dump_file && (dump_flags & TDF_DETAILS))
     752                 :            :     {
     753                 :          8 :       fprintf (dump_file, "! Disqualifying ");
     754                 :          8 :       print_generic_expr (dump_file, decl);
     755                 :          8 :       fprintf (dump_file, " - %s\n", reason);
     756                 :            :     }
     757                 :    1595020 : }
     758                 :            : 
     759                 :            : /* Return true iff the type contains a field or an element which does not allow
     760                 :            :    scalarization.  Use VISITED_TYPES to avoid re-checking already checked
     761                 :            :    (sub-)types.  */
     762                 :            : 
     763                 :            : static bool
     764                 :    5047550 : type_internals_preclude_sra_p_1 (tree type, const char **msg,
     765                 :            :                                  hash_set<tree> *visited_types)
     766                 :            : {
     767                 :    5047550 :   tree fld;
     768                 :    5047550 :   tree et;
     769                 :            : 
     770                 :    5047550 :   if (visited_types->contains (type))
     771                 :            :     return false;
     772                 :    4855940 :   visited_types->add (type);
     773                 :            : 
     774                 :    4855940 :   switch (TREE_CODE (type))
     775                 :            :     {
     776                 :    4591810 :     case RECORD_TYPE:
     777                 :    4591810 :     case UNION_TYPE:
     778                 :    4591810 :     case QUAL_UNION_TYPE:
     779                 :   61647200 :       for (fld = TYPE_FIELDS (type); fld; fld = DECL_CHAIN (fld))
     780                 :   57003300 :         if (TREE_CODE (fld) == FIELD_DECL)
     781                 :            :           {
     782                 :   10131200 :             if (TREE_CODE (fld) == FUNCTION_DECL)
     783                 :            :               continue;
     784                 :   10131200 :             tree ft = TREE_TYPE (fld);
     785                 :            : 
     786                 :   10131200 :             if (TREE_THIS_VOLATILE (fld))
     787                 :            :               {
     788                 :        608 :                 *msg = "volatile structure field";
     789                 :        608 :                 return true;
     790                 :            :               }
     791                 :   10130600 :             if (!DECL_FIELD_OFFSET (fld))
     792                 :            :               {
     793                 :          0 :                 *msg = "no structure field offset";
     794                 :          0 :                 return true;
     795                 :            :               }
     796                 :   10130600 :             if (!DECL_SIZE (fld))
     797                 :            :               {
     798                 :       6097 :                 *msg = "zero structure field size";
     799                 :       6097 :                 return true;
     800                 :            :               }
     801                 :   10124500 :             if (!tree_fits_uhwi_p (DECL_FIELD_OFFSET (fld)))
     802                 :            :               {
     803                 :          0 :                 *msg = "structure field offset not fixed";
     804                 :          0 :                 return true;
     805                 :            :               }
     806                 :   10124500 :             if (!tree_fits_uhwi_p (DECL_SIZE (fld)))
     807                 :            :               {
     808                 :          0 :                 *msg = "structure field size not fixed";
     809                 :          0 :                 return true;
     810                 :            :               }
     811                 :   10124500 :             if (!tree_fits_shwi_p (bit_position (fld)))
     812                 :            :               {
     813                 :          0 :                 *msg = "structure field size too big";
     814                 :          0 :                 return true;
     815                 :            :               }
     816                 :   10124500 :             if (AGGREGATE_TYPE_P (ft)
     817                 :   10124500 :                     && int_bit_position (fld) % BITS_PER_UNIT != 0)
     818                 :            :               {
     819                 :          0 :                 *msg = "structure field is bit field";
     820                 :          0 :                 return true;
     821                 :            :               }
     822                 :            : 
     823                 :   10124500 :             if (AGGREGATE_TYPE_P (ft)
     824                 :   10124500 :               && type_internals_preclude_sra_p_1 (ft, msg, visited_types))
     825                 :            :               return true;
     826                 :            :           }
     827                 :            : 
     828                 :            :       return false;
     829                 :            : 
     830                 :     198266 :     case ARRAY_TYPE:
     831                 :     198266 :       et = TREE_TYPE (type);
     832                 :            : 
     833                 :     198266 :       if (TYPE_VOLATILE (et))
     834                 :            :         {
     835                 :          0 :           *msg = "element type is volatile";
     836                 :          0 :           return true;
     837                 :            :         }
     838                 :            : 
     839                 :     198266 :       if (AGGREGATE_TYPE_P (et)
     840                 :     198266 :           && type_internals_preclude_sra_p_1 (et, msg, visited_types))
     841                 :        179 :         return true;
     842                 :            : 
     843                 :            :       return false;
     844                 :            : 
     845                 :            :     default:
     846                 :            :       return false;
     847                 :            :     }
     848                 :            : }
     849                 :            : 
     850                 :            : /* Return true iff the type contains a field or an element which does not allow
     851                 :            :    scalarization.  */
     852                 :            : 
     853                 :            : bool
     854                 :    3415190 : type_internals_preclude_sra_p (tree type, const char **msg)
     855                 :            : {
     856                 :    3415190 :   hash_set<tree> visited_types;
     857                 :    3415190 :   return type_internals_preclude_sra_p_1 (type, msg, &visited_types);
     858                 :            : }
     859                 :            : 
     860                 :            : 
     861                 :            : /* Allocate an access structure for BASE, OFFSET and SIZE, clear it, fill in
     862                 :            :    the three fields.  Also add it to the vector of accesses corresponding to
     863                 :            :    the base.  Finally, return the new access.  */
     864                 :            : 
     865                 :            : static struct access *
     866                 :    9350520 : create_access_1 (tree base, HOST_WIDE_INT offset, HOST_WIDE_INT size)
     867                 :            : {
     868                 :    9350520 :   struct access *access = access_pool.allocate ();
     869                 :            : 
     870                 :    9350520 :   memset (access, 0, sizeof (struct access));
     871                 :    9350520 :   access->base = base;
     872                 :    9350520 :   access->offset = offset;
     873                 :    9350520 :   access->size = size;
     874                 :            : 
     875                 :    9350520 :   base_access_vec->get_or_insert (base).safe_push (access);
     876                 :            : 
     877                 :    9350520 :   return access;
     878                 :            : }
     879                 :            : 
     880                 :            : static bool maybe_add_sra_candidate (tree);
     881                 :            : 
     882                 :            : /* Create and insert access for EXPR. Return created access, or NULL if it is
     883                 :            :    not possible.  Also scan for uses of constant pool as we go along and add
     884                 :            :    to candidates.  */
     885                 :            : 
     886                 :            : static struct access *
     887                 :   16022700 : create_access (tree expr, gimple *stmt, bool write)
     888                 :            : {
     889                 :   16022700 :   struct access *access;
     890                 :   16022700 :   poly_int64 poffset, psize, pmax_size;
     891                 :   16022700 :   tree base = expr;
     892                 :   16022700 :   bool reverse, unscalarizable_region = false;
     893                 :            : 
     894                 :   16022700 :   base = get_ref_base_and_extent (expr, &poffset, &psize, &pmax_size,
     895                 :            :                                   &reverse);
     896                 :            : 
     897                 :            :   /* For constant-pool entries, check we can substitute the constant value.  */
     898                 :   16022700 :   if (constant_decl_p (base))
     899                 :            :     {
     900                 :       2061 :       gcc_assert (!bitmap_bit_p (disqualified_constants, DECL_UID (base)));
     901                 :       2061 :       if (expr != base
     902                 :          0 :           && !is_gimple_reg_type (TREE_TYPE (expr))
     903                 :       2061 :           && dump_file && (dump_flags & TDF_DETAILS))
     904                 :            :         {
     905                 :            :           /* This occurs in Ada with accesses to ARRAY_RANGE_REFs,
     906                 :            :              and elements of multidimensional arrays (which are
     907                 :            :              multi-element arrays in their own right).  */
     908                 :          0 :           fprintf (dump_file, "Allowing non-reg-type load of part"
     909                 :            :                               " of constant-pool entry: ");
     910                 :          0 :           print_generic_expr (dump_file, expr);
     911                 :            :         }
     912                 :       2061 :       maybe_add_sra_candidate (base);
     913                 :            :     }
     914                 :            : 
     915                 :   29728500 :   if (!DECL_P (base) || !bitmap_bit_p (candidate_bitmap, DECL_UID (base)))
     916                 :    6672190 :     return NULL;
     917                 :            : 
     918                 :    9350560 :   HOST_WIDE_INT offset, size, max_size;
     919                 :    9350560 :   if (!poffset.is_constant (&offset)
     920                 :    9350560 :       || !psize.is_constant (&size)
     921                 :    9350560 :       || !pmax_size.is_constant (&max_size))
     922                 :            :     {
     923                 :            :       disqualify_candidate (base, "Encountered a polynomial-sized access.");
     924                 :            :       return NULL;
     925                 :            :     }
     926                 :            : 
     927                 :    9350560 :   if (size != max_size)
     928                 :            :     {
     929                 :     250341 :       size = max_size;
     930                 :     250341 :       unscalarizable_region = true;
     931                 :            :     }
     932                 :    9350560 :   if (size == 0)
     933                 :            :     return NULL;
     934                 :    9350560 :   if (size < 0)
     935                 :            :     {
     936                 :         35 :       disqualify_candidate (base, "Encountered an unconstrained access.");
     937                 :         35 :       return NULL;
     938                 :            :     }
     939                 :            : 
     940                 :    9350520 :   access = create_access_1 (base, offset, size);
     941                 :    9350520 :   access->expr = expr;
     942                 :    9350520 :   access->type = TREE_TYPE (expr);
     943                 :    9350520 :   access->write = write;
     944                 :    9350520 :   access->grp_unscalarizable_region = unscalarizable_region;
     945                 :    9350520 :   access->stmt = stmt;
     946                 :    9350520 :   access->reverse = reverse;
     947                 :            : 
     948                 :    9350520 :   return access;
     949                 :            : }
     950                 :            : 
     951                 :            : 
     952                 :            : /* Return true iff TYPE is scalarizable - i.e. a RECORD_TYPE or fixed-length
     953                 :            :    ARRAY_TYPE with fields that are either of gimple register types (excluding
     954                 :            :    bit-fields) or (recursively) scalarizable types.  CONST_DECL must be true if
     955                 :            :    we are considering a decl from constant pool.  If it is false, char arrays
     956                 :            :    will be refused.  */
     957                 :            : 
     958                 :            : static bool
     959                 :    3140840 : scalarizable_type_p (tree type, bool const_decl)
     960                 :            : {
     961                 :    3140840 :   if (is_gimple_reg_type (type))
     962                 :            :     return true;
     963                 :    1079200 :   if (type_contains_placeholder_p (type))
     964                 :            :     return false;
     965                 :            : 
     966                 :    1079200 :   bool have_predecessor_field = false;
     967                 :    1079200 :   HOST_WIDE_INT prev_pos = 0;
     968                 :            : 
     969                 :    1079200 :   switch (TREE_CODE (type))
     970                 :            :   {
     971                 :    1064020 :   case RECORD_TYPE:
     972                 :    7323700 :     for (tree fld = TYPE_FIELDS (type); fld; fld = DECL_CHAIN (fld))
     973                 :    6268150 :       if (TREE_CODE (fld) == FIELD_DECL)
     974                 :            :         {
     975                 :    2243920 :           tree ft = TREE_TYPE (fld);
     976                 :            : 
     977                 :    2243920 :           if (zerop (DECL_SIZE (fld)))
     978                 :       9107 :             continue;
     979                 :            : 
     980                 :    2234810 :           HOST_WIDE_INT pos = int_bit_position (fld);
     981                 :    2234810 :           if (have_predecessor_field
     982                 :    2234810 :               && pos <= prev_pos)
     983                 :            :             return false;
     984                 :            : 
     985                 :    2234810 :           have_predecessor_field = true;
     986                 :    2234810 :           prev_pos = pos;
     987                 :            : 
     988                 :    2234810 :           if (DECL_BIT_FIELD (fld))
     989                 :            :             return false;
     990                 :            : 
     991                 :    2233760 :           if (!scalarizable_type_p (ft, const_decl))
     992                 :            :             return false;
     993                 :            :         }
     994                 :            : 
     995                 :            :     return true;
     996                 :            : 
     997                 :      11022 :   case ARRAY_TYPE:
     998                 :      11022 :     {
     999                 :      11022 :       HOST_WIDE_INT min_elem_size;
    1000                 :      11022 :       if (const_decl)
    1001                 :            :         min_elem_size = 0;
    1002                 :            :       else
    1003                 :      11009 :         min_elem_size = BITS_PER_UNIT;
    1004                 :            : 
    1005                 :      11022 :       if (TYPE_DOMAIN (type) == NULL_TREE
    1006                 :      11022 :           || !tree_fits_shwi_p (TYPE_SIZE (type))
    1007                 :      11022 :           || !tree_fits_shwi_p (TYPE_SIZE (TREE_TYPE (type)))
    1008                 :      11022 :           || (tree_to_shwi (TYPE_SIZE (TREE_TYPE (type))) <= min_elem_size)
    1009                 :      17661 :           || !tree_fits_shwi_p (TYPE_MIN_VALUE (TYPE_DOMAIN (type))))
    1010                 :            :         return false;
    1011                 :       6639 :       if (tree_to_shwi (TYPE_SIZE (type)) == 0
    1012                 :       6639 :           && TYPE_MAX_VALUE (TYPE_DOMAIN (type)) == NULL_TREE)
    1013                 :            :         /* Zero-element array, should not prevent scalarization.  */
    1014                 :            :         ;
    1015                 :       6639 :       else if ((tree_to_shwi (TYPE_SIZE (type)) <= 0)
    1016                 :       6639 :                || !tree_fits_shwi_p (TYPE_MAX_VALUE (TYPE_DOMAIN (type))))
    1017                 :            :         /* Variable-length array, do not allow scalarization.  */
    1018                 :            :         return false;
    1019                 :            : 
    1020                 :       6639 :       tree elem = TREE_TYPE (type);
    1021                 :       6639 :       if (!scalarizable_type_p (elem, const_decl))
    1022                 :        132 :         return false;
    1023                 :            :       return true;
    1024                 :            :     }
    1025                 :            :   default:
    1026                 :            :     return false;
    1027                 :            :   }
    1028                 :            : }
    1029                 :            : 
    1030                 :            : /* Return true if REF has an VIEW_CONVERT_EXPR somewhere in it.  */
    1031                 :            : 
    1032                 :            : static inline bool
    1033                 :   37028100 : contains_view_convert_expr_p (const_tree ref)
    1034                 :            : {
    1035                 :   50398000 :   while (handled_component_p (ref))
    1036                 :            :     {
    1037                 :   13369900 :       if (TREE_CODE (ref) == VIEW_CONVERT_EXPR)
    1038                 :            :         return true;
    1039                 :   13369900 :       ref = TREE_OPERAND (ref, 0);
    1040                 :            :     }
    1041                 :            : 
    1042                 :            :   return false;
    1043                 :            : }
    1044                 :            : 
    1045                 :            : /* Return true if REF contains a VIEW_CONVERT_EXPR or a COMPONENT_REF with a
    1046                 :            :    bit-field field declaration.  If TYPE_CHANGING_P is non-NULL, set the bool
    1047                 :            :    it points to will be set if REF contains any of the above or a MEM_REF
    1048                 :            :    expression that effectively performs type conversion.  */
    1049                 :            : 
    1050                 :            : static bool
    1051                 :    5819100 : contains_vce_or_bfcref_p (const_tree ref, bool *type_changing_p = NULL)
    1052                 :            : {
    1053                 :    7031260 :   while (handled_component_p (ref))
    1054                 :            :     {
    1055                 :    1515370 :       if (TREE_CODE (ref) == VIEW_CONVERT_EXPR
    1056                 :    1515370 :           || (TREE_CODE (ref) == COMPONENT_REF
    1057                 :     986941 :               && DECL_BIT_FIELD (TREE_OPERAND (ref, 1))))
    1058                 :            :         {
    1059                 :     303210 :           if (type_changing_p)
    1060                 :     160151 :             *type_changing_p = true;
    1061                 :     303210 :           return true;
    1062                 :            :         }
    1063                 :    1212160 :       ref = TREE_OPERAND (ref, 0);
    1064                 :            :     }
    1065                 :            : 
    1066                 :    5515890 :   if (!type_changing_p
    1067                 :    2737660 :       || TREE_CODE (ref) != MEM_REF
    1068                 :    5542920 :       || TREE_CODE (TREE_OPERAND (ref, 0)) != ADDR_EXPR)
    1069                 :            :     return false;
    1070                 :            : 
    1071                 :      27027 :   tree mem = TREE_OPERAND (TREE_OPERAND (ref, 0), 0);
    1072                 :      27027 :   if (TYPE_MAIN_VARIANT (TREE_TYPE (ref))
    1073                 :      27027 :       != TYPE_MAIN_VARIANT (TREE_TYPE (mem)))
    1074                 :      17938 :     *type_changing_p = true;
    1075                 :            : 
    1076                 :            :   return false;
    1077                 :            : }
    1078                 :            : 
    1079                 :            : /* Search the given tree for a declaration by skipping handled components and
    1080                 :            :    exclude it from the candidates.  */
    1081                 :            : 
    1082                 :            : static void
    1083                 :      12576 : disqualify_base_of_expr (tree t, const char *reason)
    1084                 :            : {
    1085                 :      12576 :   t = get_base_address (t);
    1086                 :      12576 :   if (t && DECL_P (t))
    1087                 :       9502 :     disqualify_candidate (t, reason);
    1088                 :      12576 : }
    1089                 :            : 
    1090                 :            : /* Scan expression EXPR and create access structures for all accesses to
    1091                 :            :    candidates for scalarization.  Return the created access or NULL if none is
    1092                 :            :    created.  */
    1093                 :            : 
    1094                 :            : static struct access *
    1095                 :   37028100 : build_access_from_expr_1 (tree expr, gimple *stmt, bool write)
    1096                 :            : {
    1097                 :   37028100 :   struct access *ret = NULL;
    1098                 :   37028100 :   bool partial_ref;
    1099                 :            : 
    1100                 :   37028100 :   if (TREE_CODE (expr) == BIT_FIELD_REF
    1101                 :   37028100 :       || TREE_CODE (expr) == IMAGPART_EXPR
    1102                 :   37007800 :       || TREE_CODE (expr) == REALPART_EXPR)
    1103                 :            :     {
    1104                 :      27292 :       expr = TREE_OPERAND (expr, 0);
    1105                 :      27292 :       partial_ref = true;
    1106                 :            :     }
    1107                 :            :   else
    1108                 :            :     partial_ref = false;
    1109                 :            : 
    1110                 :   37028100 :   if (storage_order_barrier_p (expr))
    1111                 :            :     {
    1112                 :          0 :       disqualify_base_of_expr (expr, "storage order barrier.");
    1113                 :          0 :       return NULL;
    1114                 :            :     }
    1115                 :            : 
    1116                 :            :   /* We need to dive through V_C_Es in order to get the size of its parameter
    1117                 :            :      and not the result type.  Ada produces such statements.  We are also
    1118                 :            :      capable of handling the topmost V_C_E but not any of those buried in other
    1119                 :            :      handled components.  */
    1120                 :   37028100 :   if (TREE_CODE (expr) == VIEW_CONVERT_EXPR)
    1121                 :     230037 :     expr = TREE_OPERAND (expr, 0);
    1122                 :            : 
    1123                 :   37028100 :   if (contains_view_convert_expr_p (expr))
    1124                 :            :     {
    1125                 :         54 :       disqualify_base_of_expr (expr, "V_C_E under a different handled "
    1126                 :            :                                "component.");
    1127                 :         54 :       return NULL;
    1128                 :            :     }
    1129                 :   37028000 :   if (TREE_THIS_VOLATILE (expr))
    1130                 :            :     {
    1131                 :      12406 :       disqualify_base_of_expr (expr, "part of a volatile reference.");
    1132                 :      12406 :       return NULL;
    1133                 :            :     }
    1134                 :            : 
    1135                 :   37015600 :   switch (TREE_CODE (expr))
    1136                 :            :     {
    1137                 :    2062970 :     case MEM_REF:
    1138                 :    2062970 :       if (TREE_CODE (TREE_OPERAND (expr, 0)) != ADDR_EXPR)
    1139                 :            :         return NULL;
    1140                 :            :       /* fall through */
    1141                 :   16022700 :     case VAR_DECL:
    1142                 :   16022700 :     case PARM_DECL:
    1143                 :   16022700 :     case RESULT_DECL:
    1144                 :   16022700 :     case COMPONENT_REF:
    1145                 :   16022700 :     case ARRAY_REF:
    1146                 :   16022700 :     case ARRAY_RANGE_REF:
    1147                 :   16022700 :       ret = create_access (expr, stmt, write);
    1148                 :   16022700 :       break;
    1149                 :            : 
    1150                 :            :     default:
    1151                 :            :       break;
    1152                 :            :     }
    1153                 :            : 
    1154                 :   35940000 :   if (write && partial_ref && ret)
    1155                 :       2143 :     ret->grp_partial_lhs = 1;
    1156                 :            : 
    1157                 :            :   return ret;
    1158                 :            : }
    1159                 :            : 
    1160                 :            : /* Scan expression EXPR and create access structures for all accesses to
    1161                 :            :    candidates for scalarization.  Return true if any access has been inserted.
    1162                 :            :    STMT must be the statement from which the expression is taken, WRITE must be
    1163                 :            :    true if the expression is a store and false otherwise. */
    1164                 :            : 
    1165                 :            : static bool
    1166                 :    9125750 : build_access_from_expr (tree expr, gimple *stmt, bool write)
    1167                 :            : {
    1168                 :    9125750 :   struct access *access;
    1169                 :            : 
    1170                 :    9125750 :   access = build_access_from_expr_1 (expr, stmt, write);
    1171                 :    9125750 :   if (access)
    1172                 :            :     {
    1173                 :            :       /* This means the aggregate is accesses as a whole in a way other than an
    1174                 :            :          assign statement and thus cannot be removed even if we had a scalar
    1175                 :            :          replacement for everything.  */
    1176                 :    1398300 :       if (cannot_scalarize_away_bitmap)
    1177                 :    1398300 :         bitmap_set_bit (cannot_scalarize_away_bitmap, DECL_UID (access->base));
    1178                 :    1398300 :       return true;
    1179                 :            :     }
    1180                 :            :   return false;
    1181                 :            : }
    1182                 :            : 
    1183                 :            : /* Return the single non-EH successor edge of BB or NULL if there is none or
    1184                 :            :    more than one.  */
    1185                 :            : 
    1186                 :            : static edge
    1187                 :    1123230 : single_non_eh_succ (basic_block bb)
    1188                 :            : {
    1189                 :    1123230 :   edge e, res = NULL;
    1190                 :    1123230 :   edge_iterator ei;
    1191                 :            : 
    1192                 :    3368270 :   FOR_EACH_EDGE (e, ei, bb->succs)
    1193                 :    2245050 :     if (!(e->flags & EDGE_EH))
    1194                 :            :       {
    1195                 :    1123120 :         if (res)
    1196                 :            :           return NULL;
    1197                 :            :         res = e;
    1198                 :            :       }
    1199                 :            : 
    1200                 :            :   return res;
    1201                 :            : }
    1202                 :            : 
    1203                 :            : /* Disqualify LHS and RHS for scalarization if STMT has to terminate its BB and
    1204                 :            :    there is no alternative spot where to put statements SRA might need to
    1205                 :            :    generate after it.  The spot we are looking for is an edge leading to a
    1206                 :            :    single non-EH successor, if it exists and is indeed single.  RHS may be
    1207                 :            :    NULL, in that case ignore it.  */
    1208                 :            : 
    1209                 :            : static bool
    1210                 :   15558500 : disqualify_if_bad_bb_terminating_stmt (gimple *stmt, tree lhs, tree rhs)
    1211                 :            : {
    1212                 :   15558500 :   if (stmt_ends_bb_p (stmt))
    1213                 :            :     {
    1214                 :    1047750 :       if (single_non_eh_succ (gimple_bb (stmt)))
    1215                 :            :         return false;
    1216                 :            : 
    1217                 :        116 :       disqualify_base_of_expr (lhs, "LHS of a throwing stmt.");
    1218                 :        116 :       if (rhs)
    1219                 :          0 :         disqualify_base_of_expr (rhs, "RHS of a throwing stmt.");
    1220                 :        116 :       return true;
    1221                 :            :     }
    1222                 :            :   return false;
    1223                 :            : }
    1224                 :            : 
    1225                 :            : /* Return true if the nature of BASE is such that it contains data even if
    1226                 :            :    there is no write to it in the function.  */
    1227                 :            : 
    1228                 :            : static bool
    1229                 :    2596930 : comes_initialized_p (tree base)
    1230                 :            : {
    1231                 :          0 :   return TREE_CODE (base) == PARM_DECL || constant_decl_p (base);
    1232                 :            : }
    1233                 :            : 
    1234                 :            : /* Scan expressions occurring in STMT, create access structures for all accesses
    1235                 :            :    to candidates for scalarization and remove those candidates which occur in
    1236                 :            :    statements or expressions that prevent them from being split apart.  Return
    1237                 :            :    true if any access has been inserted.  */
    1238                 :            : 
    1239                 :            : static bool
    1240                 :   20817900 : build_accesses_from_assign (gimple *stmt)
    1241                 :            : {
    1242                 :   20817900 :   tree lhs, rhs;
    1243                 :   20817900 :   struct access *lacc, *racc;
    1244                 :            : 
    1245                 :   20817900 :   if (!gimple_assign_single_p (stmt)
    1246                 :            :       /* Scope clobbers don't influence scalarization.  */
    1247                 :   15988700 :       || gimple_clobber_p (stmt))
    1248                 :            :     return false;
    1249                 :            : 
    1250                 :   13951200 :   lhs = gimple_assign_lhs (stmt);
    1251                 :   13951200 :   rhs = gimple_assign_rhs1 (stmt);
    1252                 :            : 
    1253                 :   13951200 :   if (disqualify_if_bad_bb_terminating_stmt (stmt, lhs, rhs))
    1254                 :            :     return false;
    1255                 :            : 
    1256                 :   13951200 :   racc = build_access_from_expr_1 (rhs, stmt, false);
    1257                 :   13951200 :   lacc = build_access_from_expr_1 (lhs, stmt, true);
    1258                 :            : 
    1259                 :   13951200 :   if (lacc)
    1260                 :            :     {
    1261                 :    4177280 :       lacc->grp_assignment_write = 1;
    1262                 :    4177280 :       if (storage_order_barrier_p (rhs))
    1263                 :          0 :         lacc->grp_unscalarizable_region = 1;
    1264                 :            : 
    1265                 :    4177280 :       if (should_scalarize_away_bitmap && !is_gimple_reg_type (lacc->type))
    1266                 :            :         {
    1267                 :    1427310 :           bool type_changing_p = false;
    1268                 :    1427310 :           contains_vce_or_bfcref_p (lhs, &type_changing_p);
    1269                 :    1427310 :           if (type_changing_p)
    1270                 :      24310 :             bitmap_set_bit (cannot_scalarize_away_bitmap,
    1271                 :      12155 :                             DECL_UID (lacc->base));
    1272                 :            :         }
    1273                 :            :     }
    1274                 :            : 
    1275                 :   13951200 :   if (racc)
    1276                 :            :     {
    1277                 :    3774930 :       racc->grp_assignment_read = 1;
    1278                 :    3774930 :       if (should_scalarize_away_bitmap && !is_gimple_reg_type (racc->type))
    1279                 :            :         {
    1280                 :    1470510 :           bool type_changing_p = false;
    1281                 :    1470510 :           contains_vce_or_bfcref_p (rhs, &type_changing_p);
    1282                 :            : 
    1283                 :    2775080 :           if (type_changing_p || gimple_has_volatile_ops (stmt))
    1284                 :     332206 :             bitmap_set_bit (cannot_scalarize_away_bitmap,
    1285                 :     166103 :                             DECL_UID (racc->base));
    1286                 :            :           else
    1287                 :    2608810 :             bitmap_set_bit (should_scalarize_away_bitmap,
    1288                 :    1304400 :                             DECL_UID (racc->base));
    1289                 :            :         }
    1290                 :    3774930 :       if (storage_order_barrier_p (lhs))
    1291                 :          0 :         racc->grp_unscalarizable_region = 1;
    1292                 :            :     }
    1293                 :            : 
    1294                 :   13951200 :   if (lacc && racc
    1295                 :    1064080 :       && (sra_mode == SRA_MODE_EARLY_INTRA || sra_mode == SRA_MODE_INTRA)
    1296                 :    1064080 :       && !lacc->grp_unscalarizable_region
    1297                 :    1063700 :       && !racc->grp_unscalarizable_region
    1298                 :    1063190 :       && AGGREGATE_TYPE_P (TREE_TYPE (lhs))
    1299                 :    1063190 :       && lacc->size == racc->size
    1300                 :   15014300 :       && useless_type_conversion_p (lacc->type, racc->type))
    1301                 :            :     {
    1302                 :     909118 :       struct assign_link *link;
    1303                 :            : 
    1304                 :     909118 :       link = assign_link_pool.allocate ();
    1305                 :     909118 :       memset (link, 0, sizeof (struct assign_link));
    1306                 :            : 
    1307                 :     909118 :       link->lacc = lacc;
    1308                 :     909118 :       link->racc = racc;
    1309                 :     909118 :       add_link_to_rhs (racc, link);
    1310                 :     909118 :       add_link_to_lhs (lacc, link);
    1311                 :     909118 :       add_access_to_rhs_work_queue (racc);
    1312                 :     909118 :       add_access_to_lhs_work_queue (lacc);
    1313                 :            : 
    1314                 :            :       /* Let's delay marking the areas as written until propagation of accesses
    1315                 :            :          across link, unless the nature of rhs tells us that its data comes
    1316                 :            :          from elsewhere.  */
    1317                 :     909118 :       if (!comes_initialized_p (racc->base))
    1318                 :     843451 :         lacc->write = false;
    1319                 :            :     }
    1320                 :            : 
    1321                 :   13951200 :   return lacc || racc;
    1322                 :            : }
    1323                 :            : 
    1324                 :            : /* Callback of walk_stmt_load_store_addr_ops visit_addr used to determine
    1325                 :            :    GIMPLE_ASM operands with memory constrains which cannot be scalarized.  */
    1326                 :            : 
    1327                 :            : static bool
    1328                 :        798 : asm_visit_addr (gimple *, tree op, tree, void *)
    1329                 :            : {
    1330                 :        798 :   op = get_base_address (op);
    1331                 :        798 :   if (op
    1332                 :        798 :       && DECL_P (op))
    1333                 :        769 :     disqualify_candidate (op, "Non-scalarizable GIMPLE_ASM operand.");
    1334                 :            : 
    1335                 :        798 :   return false;
    1336                 :            : }
    1337                 :            : 
    1338                 :            : /* Scan function and look for interesting expressions and create access
    1339                 :            :    structures for them.  Return true iff any access is created.  */
    1340                 :            : 
    1341                 :            : static bool
    1342                 :     492288 : scan_function (void)
    1343                 :            : {
    1344                 :     492288 :   basic_block bb;
    1345                 :     492288 :   bool ret = false;
    1346                 :            : 
    1347                 :    8574820 :   FOR_EACH_BB_FN (bb, cfun)
    1348                 :            :     {
    1349                 :    8082540 :       gimple_stmt_iterator gsi;
    1350                 :   77187400 :       for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
    1351                 :            :         {
    1352                 :   61022400 :           gimple *stmt = gsi_stmt (gsi);
    1353                 :   61022400 :           tree t;
    1354                 :   61022400 :           unsigned i;
    1355                 :            : 
    1356                 :   61022400 :           switch (gimple_code (stmt))
    1357                 :            :             {
    1358                 :     487040 :             case GIMPLE_RETURN:
    1359                 :     487040 :               t = gimple_return_retval (as_a <greturn *> (stmt));
    1360                 :     487040 :               if (t != NULL_TREE)
    1361                 :     285989 :                 ret |= build_access_from_expr (t, stmt, false);
    1362                 :            :               break;
    1363                 :            : 
    1364                 :   20817900 :             case GIMPLE_ASSIGN:
    1365                 :   20817900 :               ret |= build_accesses_from_assign (stmt);
    1366                 :   20817900 :               break;
    1367                 :            : 
    1368                 :            :             case GIMPLE_CALL:
    1369                 :   10899500 :               for (i = 0; i < gimple_call_num_args (stmt); i++)
    1370                 :    7221450 :                 ret |= build_access_from_expr (gimple_call_arg (stmt, i),
    1371                 :            :                                                stmt, false);
    1372                 :            : 
    1373                 :    3678010 :               t = gimple_call_lhs (stmt);
    1374                 :    3678010 :               if (t && !disqualify_if_bad_bb_terminating_stmt (stmt, t, NULL))
    1375                 :    1607240 :                 ret |= build_access_from_expr (t, stmt, true);
    1376                 :            :               break;
    1377                 :            : 
    1378                 :       6006 :             case GIMPLE_ASM:
    1379                 :       6006 :               {
    1380                 :       6006 :                 gasm *asm_stmt = as_a <gasm *> (stmt);
    1381                 :       6006 :                 walk_stmt_load_store_addr_ops (asm_stmt, NULL, NULL, NULL,
    1382                 :            :                                                asm_visit_addr);
    1383                 :      12799 :                 for (i = 0; i < gimple_asm_ninputs (asm_stmt); i++)
    1384                 :            :                   {
    1385                 :       6793 :                     t = TREE_VALUE (gimple_asm_input_op (asm_stmt, i));
    1386                 :       6793 :                     ret |= build_access_from_expr (t, asm_stmt, false);
    1387                 :            :                   }
    1388                 :      10288 :                 for (i = 0; i < gimple_asm_noutputs (asm_stmt); i++)
    1389                 :            :                   {
    1390                 :       4282 :                     t = TREE_VALUE (gimple_asm_output_op (asm_stmt, i));
    1391                 :       4282 :                     ret |= build_access_from_expr (t, asm_stmt, true);
    1392                 :            :                   }
    1393                 :            :               }
    1394                 :            :               break;
    1395                 :            : 
    1396                 :            :             default:
    1397                 :            :               break;
    1398                 :            :             }
    1399                 :            :         }
    1400                 :            :     }
    1401                 :            : 
    1402                 :     492288 :   return ret;
    1403                 :            : }
    1404                 :            : 
    1405                 :            : /* Helper of QSORT function. There are pointers to accesses in the array.  An
    1406                 :            :    access is considered smaller than another if it has smaller offset or if the
    1407                 :            :    offsets are the same but is size is bigger. */
    1408                 :            : 
    1409                 :            : static int
    1410                 :   63651900 : compare_access_positions (const void *a, const void *b)
    1411                 :            : {
    1412                 :   63651900 :   const access_p *fp1 = (const access_p *) a;
    1413                 :   63651900 :   const access_p *fp2 = (const access_p *) b;
    1414                 :   63651900 :   const access_p f1 = *fp1;
    1415                 :   63651900 :   const access_p f2 = *fp2;
    1416                 :            : 
    1417                 :   63651900 :   if (f1->offset != f2->offset)
    1418                 :   52937500 :     return f1->offset < f2->offset ? -1 : 1;
    1419                 :            : 
    1420                 :   27658000 :   if (f1->size == f2->size)
    1421                 :            :     {
    1422                 :   18468100 :       if (f1->type == f2->type)
    1423                 :            :         return 0;
    1424                 :            :       /* Put any non-aggregate type before any aggregate type.  */
    1425                 :    2274280 :       else if (!is_gimple_reg_type (f1->type)
    1426                 :    2274280 :           && is_gimple_reg_type (f2->type))
    1427                 :            :         return 1;
    1428                 :    1636440 :       else if (is_gimple_reg_type (f1->type)
    1429                 :    1636440 :                && !is_gimple_reg_type (f2->type))
    1430                 :            :         return -1;
    1431                 :            :       /* Put any complex or vector type before any other scalar type.  */
    1432                 :     790317 :       else if (TREE_CODE (f1->type) != COMPLEX_TYPE
    1433                 :     790317 :                && TREE_CODE (f1->type) != VECTOR_TYPE
    1434                 :     750654 :                && (TREE_CODE (f2->type) == COMPLEX_TYPE
    1435                 :     750654 :                    || TREE_CODE (f2->type) == VECTOR_TYPE))
    1436                 :            :         return 1;
    1437                 :     762857 :       else if ((TREE_CODE (f1->type) == COMPLEX_TYPE
    1438                 :            :                 || TREE_CODE (f1->type) == VECTOR_TYPE)
    1439                 :      39663 :                && TREE_CODE (f2->type) != COMPLEX_TYPE
    1440                 :      37108 :                && TREE_CODE (f2->type) != VECTOR_TYPE)
    1441                 :            :         return -1;
    1442                 :            :       /* Put any integral type before any non-integral type.  When splicing, we
    1443                 :            :          make sure that those with insufficient precision and occupying the
    1444                 :            :          same space are not scalarized.  */
    1445                 :     725948 :       else if (INTEGRAL_TYPE_P (f1->type)
    1446                 :     183986 :                && !INTEGRAL_TYPE_P (f2->type))
    1447                 :            :         return -1;
    1448                 :     620604 :       else if (!INTEGRAL_TYPE_P (f1->type)
    1449                 :     541962 :                && INTEGRAL_TYPE_P (f2->type))
    1450                 :            :         return 1;
    1451                 :            :       /* Put the integral type with the bigger precision first.  */
    1452                 :     513315 :       else if (INTEGRAL_TYPE_P (f1->type)
    1453                 :      78642 :                && INTEGRAL_TYPE_P (f2->type)
    1454                 :     591957 :                && (TYPE_PRECISION (f2->type) != TYPE_PRECISION (f1->type)))
    1455                 :       9406 :         return TYPE_PRECISION (f2->type) - TYPE_PRECISION (f1->type);
    1456                 :            :       /* Stabilize the sort.  */
    1457                 :     503909 :       return TYPE_UID (f1->type) - TYPE_UID (f2->type);
    1458                 :            :     }
    1459                 :            : 
    1460                 :            :   /* We want the bigger accesses first, thus the opposite operator in the next
    1461                 :            :      line: */
    1462                 :    9189980 :   return f1->size > f2->size ? -1 : 1;
    1463                 :            : }
    1464                 :            : 
    1465                 :            : 
    1466                 :            : /* Append a name of the declaration to the name obstack.  A helper function for
    1467                 :            :    make_fancy_name.  */
    1468                 :            : 
    1469                 :            : static void
    1470                 :    1372790 : make_fancy_decl_name (tree decl)
    1471                 :            : {
    1472                 :    1372790 :   char buffer[32];
    1473                 :            : 
    1474                 :    1372790 :   tree name = DECL_NAME (decl);
    1475                 :    1372790 :   if (name)
    1476                 :    1320350 :     obstack_grow (&name_obstack, IDENTIFIER_POINTER (name),
    1477                 :            :                   IDENTIFIER_LENGTH (name));
    1478                 :            :   else
    1479                 :            :     {
    1480                 :      52437 :       sprintf (buffer, "D%u", DECL_UID (decl));
    1481                 :      52437 :       obstack_grow (&name_obstack, buffer, strlen (buffer));
    1482                 :            :     }
    1483                 :    1372790 : }
    1484                 :            : 
    1485                 :            : /* Helper for make_fancy_name.  */
    1486                 :            : 
    1487                 :            : static void
    1488                 :    1551330 : make_fancy_name_1 (tree expr)
    1489                 :            : {
    1490                 :    1551330 :   char buffer[32];
    1491                 :    1551330 :   tree index;
    1492                 :            : 
    1493                 :    1551330 :   if (DECL_P (expr))
    1494                 :            :     {
    1495                 :     660445 :       make_fancy_decl_name (expr);
    1496                 :     660445 :       return;
    1497                 :            :     }
    1498                 :            : 
    1499                 :     890889 :   switch (TREE_CODE (expr))
    1500                 :            :     {
    1501                 :     712344 :     case COMPONENT_REF:
    1502                 :     712344 :       make_fancy_name_1 (TREE_OPERAND (expr, 0));
    1503                 :     712344 :       obstack_1grow (&name_obstack, '$');
    1504                 :     712344 :       make_fancy_decl_name (TREE_OPERAND (expr, 1));
    1505                 :     712344 :       break;
    1506                 :            : 
    1507                 :      32445 :     case ARRAY_REF:
    1508                 :      32445 :       make_fancy_name_1 (TREE_OPERAND (expr, 0));
    1509                 :      32445 :       obstack_1grow (&name_obstack, '$');
    1510                 :            :       /* Arrays with only one element may not have a constant as their
    1511                 :            :          index. */
    1512                 :      32445 :       index = TREE_OPERAND (expr, 1);
    1513                 :      32445 :       if (TREE_CODE (index) != INTEGER_CST)
    1514                 :            :         break;
    1515                 :      32385 :       sprintf (buffer, HOST_WIDE_INT_PRINT_DEC, TREE_INT_CST_LOW (index));
    1516                 :      32385 :       obstack_grow (&name_obstack, buffer, strlen (buffer));
    1517                 :      32385 :       break;
    1518                 :            : 
    1519                 :      73000 :     case ADDR_EXPR:
    1520                 :      73000 :       make_fancy_name_1 (TREE_OPERAND (expr, 0));
    1521                 :      73000 :       break;
    1522                 :            : 
    1523                 :      73000 :     case MEM_REF:
    1524                 :      73000 :       make_fancy_name_1 (TREE_OPERAND (expr, 0));
    1525                 :      73000 :       if (!integer_zerop (TREE_OPERAND (expr, 1)))
    1526                 :            :         {
    1527                 :      18613 :           obstack_1grow (&name_obstack, '$');
    1528                 :      37226 :           sprintf (buffer, HOST_WIDE_INT_PRINT_DEC,
    1529                 :      18613 :                    TREE_INT_CST_LOW (TREE_OPERAND (expr, 1)));
    1530                 :      18613 :           obstack_grow (&name_obstack, buffer, strlen (buffer));
    1531                 :            :         }
    1532                 :            :       break;
    1533                 :            : 
    1534                 :          0 :     case BIT_FIELD_REF:
    1535                 :          0 :     case REALPART_EXPR:
    1536                 :          0 :     case IMAGPART_EXPR:
    1537                 :          0 :       gcc_unreachable ();       /* we treat these as scalars.  */
    1538                 :            :       break;
    1539                 :            :     default:
    1540                 :            :       break;
    1541                 :            :     }
    1542                 :            : }
    1543                 :            : 
    1544                 :            : /* Create a human readable name for replacement variable of ACCESS.  */
    1545                 :            : 
    1546                 :            : static char *
    1547                 :     660545 : make_fancy_name (tree expr)
    1548                 :            : {
    1549                 :     660545 :   make_fancy_name_1 (expr);
    1550                 :     660545 :   obstack_1grow (&name_obstack, '\0');
    1551                 :     660545 :   return XOBFINISH (&name_obstack, char *);
    1552                 :            : }
    1553                 :            : 
    1554                 :            : /* Construct a MEM_REF that would reference a part of aggregate BASE of type
    1555                 :            :    EXP_TYPE at the given OFFSET and with storage order REVERSE.  If BASE is
    1556                 :            :    something for which get_addr_base_and_unit_offset returns NULL, gsi must
    1557                 :            :    be non-NULL and is used to insert new statements either before or below
    1558                 :            :    the current one as specified by INSERT_AFTER.  This function is not capable
    1559                 :            :    of handling bitfields.  */
    1560                 :            : 
    1561                 :            : tree
    1562                 :     856029 : build_ref_for_offset (location_t loc, tree base, poly_int64 offset,
    1563                 :            :                       bool reverse, tree exp_type, gimple_stmt_iterator *gsi,
    1564                 :            :                       bool insert_after)
    1565                 :            : {
    1566                 :     856029 :   tree prev_base = base;
    1567                 :     856029 :   tree off;
    1568                 :     856029 :   tree mem_ref;
    1569                 :     856029 :   poly_int64 base_offset;
    1570                 :     856029 :   unsigned HOST_WIDE_INT misalign;
    1571                 :     856029 :   unsigned int align;
    1572                 :            : 
    1573                 :            :   /* Preserve address-space information.  */
    1574                 :     856029 :   addr_space_t as = TYPE_ADDR_SPACE (TREE_TYPE (base));
    1575                 :     856029 :   if (as != TYPE_ADDR_SPACE (exp_type))
    1576                 :          2 :     exp_type = build_qualified_type (exp_type,
    1577                 :          2 :                                      TYPE_QUALS (exp_type)
    1578                 :          2 :                                      | ENCODE_QUAL_ADDR_SPACE (as));
    1579                 :            : 
    1580                 :     856029 :   poly_int64 byte_offset = exact_div (offset, BITS_PER_UNIT);
    1581                 :     856029 :   get_object_alignment_1 (base, &align, &misalign);
    1582                 :     856029 :   base = get_addr_base_and_unit_offset (base, &base_offset);
    1583                 :            : 
    1584                 :            :   /* get_addr_base_and_unit_offset returns NULL for references with a variable
    1585                 :            :      offset such as array[var_index].  */
    1586                 :     856029 :   if (!base)
    1587                 :            :     {
    1588                 :       3344 :       gassign *stmt;
    1589                 :       3344 :       tree tmp, addr;
    1590                 :            : 
    1591                 :       3344 :       gcc_checking_assert (gsi);
    1592                 :       3344 :       tmp = make_ssa_name (build_pointer_type (TREE_TYPE (prev_base)));
    1593                 :       3344 :       addr = build_fold_addr_expr (unshare_expr (prev_base));
    1594                 :       3344 :       STRIP_USELESS_TYPE_CONVERSION (addr);
    1595                 :       3344 :       stmt = gimple_build_assign (tmp, addr);
    1596                 :       3344 :       gimple_set_location (stmt, loc);
    1597                 :       3344 :       if (insert_after)
    1598                 :       1663 :         gsi_insert_after (gsi, stmt, GSI_NEW_STMT);
    1599                 :            :       else
    1600                 :       1681 :         gsi_insert_before (gsi, stmt, GSI_SAME_STMT);
    1601                 :            : 
    1602                 :       3344 :       off = build_int_cst (reference_alias_ptr_type (prev_base), byte_offset);
    1603                 :       3344 :       base = tmp;
    1604                 :            :     }
    1605                 :     852685 :   else if (TREE_CODE (base) == MEM_REF)
    1606                 :            :     {
    1607                 :      56875 :       off = build_int_cst (TREE_TYPE (TREE_OPERAND (base, 1)),
    1608                 :            :                            base_offset + byte_offset);
    1609                 :      56875 :       off = int_const_binop (PLUS_EXPR, TREE_OPERAND (base, 1), off);
    1610                 :      56875 :       base = unshare_expr (TREE_OPERAND (base, 0));
    1611                 :            :     }
    1612                 :            :   else
    1613                 :            :     {
    1614                 :     795810 :       off = build_int_cst (reference_alias_ptr_type (prev_base),
    1615                 :            :                            base_offset + byte_offset);
    1616                 :     795810 :       base = build_fold_addr_expr (unshare_expr (base));
    1617                 :            :     }
    1618                 :            : 
    1619                 :     856029 :   unsigned int align_bound = known_alignment (misalign + offset);
    1620                 :     856029 :   if (align_bound != 0)
    1621                 :     611513 :     align = MIN (align, align_bound);
    1622                 :     856029 :   if (align != TYPE_ALIGN (exp_type))
    1623                 :     101339 :     exp_type = build_aligned_type (exp_type, align);
    1624                 :            : 
    1625                 :     856029 :   mem_ref = fold_build2_loc (loc, MEM_REF, exp_type, base, off);
    1626                 :     856029 :   REF_REVERSE_STORAGE_ORDER (mem_ref) = reverse;
    1627                 :     856029 :   if (TREE_THIS_VOLATILE (prev_base))
    1628                 :          6 :     TREE_THIS_VOLATILE (mem_ref) = 1;
    1629                 :     856029 :   if (TREE_SIDE_EFFECTS (prev_base))
    1630                 :          6 :     TREE_SIDE_EFFECTS (mem_ref) = 1;
    1631                 :     856029 :   return mem_ref;
    1632                 :            : }
    1633                 :            : 
    1634                 :            : /* Construct and return a memory reference that is equal to a portion of
    1635                 :            :    MODEL->expr but is based on BASE.  If this cannot be done, return NULL.  */
    1636                 :            : 
    1637                 :            : static tree
    1638                 :    1902820 : build_reconstructed_reference (location_t, tree base, struct access *model)
    1639                 :            : {
    1640                 :    1902820 :   tree expr = model->expr, prev_expr = NULL;
    1641                 :    4129080 :   while (!types_compatible_p (TREE_TYPE (expr), TREE_TYPE (base)))
    1642                 :            :     {
    1643                 :    2321810 :       if (!handled_component_p (expr))
    1644                 :            :         return NULL_TREE;
    1645                 :    2226270 :       prev_expr = expr;
    1646                 :    2226270 :       expr = TREE_OPERAND (expr, 0);
    1647                 :            :     }
    1648                 :            : 
    1649                 :            :   /* Guard against broken VIEW_CONVERT_EXPRs...  */
    1650                 :    1807270 :   if (!prev_expr)
    1651                 :            :     return NULL_TREE;
    1652                 :            : 
    1653                 :    1807270 :   TREE_OPERAND (prev_expr, 0) = base;
    1654                 :    1807270 :   tree ref = unshare_expr (model->expr);
    1655                 :    1807270 :   TREE_OPERAND (prev_expr, 0) = expr;
    1656                 :    1807270 :   return ref;
    1657                 :            : }
    1658                 :            : 
    1659                 :            : /* Construct a memory reference to a part of an aggregate BASE at the given
    1660                 :            :    OFFSET and of the same type as MODEL.  In case this is a reference to a
    1661                 :            :    bit-field, the function will replicate the last component_ref of model's
    1662                 :            :    expr to access it.  GSI and INSERT_AFTER have the same meaning as in
    1663                 :            :    build_ref_for_offset.  */
    1664                 :            : 
    1665                 :            : static tree
    1666                 :    2614860 : build_ref_for_model (location_t loc, tree base, HOST_WIDE_INT offset,
    1667                 :            :                      struct access *model, gimple_stmt_iterator *gsi,
    1668                 :            :                      bool insert_after)
    1669                 :            : {
    1670                 :    2614860 :   if (TREE_CODE (model->expr) == COMPONENT_REF
    1671                 :    2614860 :       && DECL_BIT_FIELD (TREE_OPERAND (model->expr, 1)))
    1672                 :            :     {
    1673                 :            :       /* This access represents a bit-field.  */
    1674                 :       6674 :       tree t, exp_type, fld = TREE_OPERAND (model->expr, 1);
    1675                 :            : 
    1676                 :       6674 :       offset -= int_bit_position (fld);
    1677                 :       6674 :       exp_type = TREE_TYPE (TREE_OPERAND (model->expr, 0));
    1678                 :      13348 :       t = build_ref_for_offset (loc, base, offset, model->reverse, exp_type,
    1679                 :       6674 :                                 gsi, insert_after);
    1680                 :            :       /* The flag will be set on the record type.  */
    1681                 :       6674 :       REF_REVERSE_STORAGE_ORDER (t) = 0;
    1682                 :       6674 :       return fold_build3_loc (loc, COMPONENT_REF, TREE_TYPE (fld), t, fld,
    1683                 :       6674 :                               NULL_TREE);
    1684                 :            :     }
    1685                 :            :   else
    1686                 :            :     {
    1687                 :    2608180 :       tree res;
    1688                 :    2608180 :       if (model->grp_same_access_path
    1689                 :    1902840 :           && !TREE_THIS_VOLATILE (base)
    1690                 :    1902840 :           && (TYPE_ADDR_SPACE (TREE_TYPE (base))
    1691                 :    1902840 :               == TYPE_ADDR_SPACE (TREE_TYPE (model->expr)))
    1692                 :    1902840 :           && offset <= model->offset
    1693                 :            :           /* build_reconstructed_reference can still fail if we have already
    1694                 :            :              massaged BASE because of another type incompatibility.  */
    1695                 :    4511000 :           && (res = build_reconstructed_reference (loc, base, model)))
    1696                 :            :         return res;
    1697                 :            :       else
    1698                 :     800910 :         return build_ref_for_offset (loc, base, offset, model->reverse,
    1699                 :     800910 :                                      model->type, gsi, insert_after);
    1700                 :            :     }
    1701                 :            : }
    1702                 :            : 
    1703                 :            : /* Attempt to build a memory reference that we could but into a gimple
    1704                 :            :    debug_bind statement.  Similar to build_ref_for_model but punts if it has to
    1705                 :            :    create statements and return s NULL instead.  This function also ignores
    1706                 :            :    alignment issues and so its results should never end up in non-debug
    1707                 :            :    statements.  */
    1708                 :            : 
    1709                 :            : static tree
    1710                 :        247 : build_debug_ref_for_model (location_t loc, tree base, HOST_WIDE_INT offset,
    1711                 :            :                            struct access *model)
    1712                 :            : {
    1713                 :        247 :   poly_int64 base_offset;
    1714                 :        247 :   tree off;
    1715                 :            : 
    1716                 :        247 :   if (TREE_CODE (model->expr) == COMPONENT_REF
    1717                 :        247 :       && DECL_BIT_FIELD (TREE_OPERAND (model->expr, 1)))
    1718                 :            :     return NULL_TREE;
    1719                 :            : 
    1720                 :        247 :   base = get_addr_base_and_unit_offset (base, &base_offset);
    1721                 :        247 :   if (!base)
    1722                 :            :     return NULL_TREE;
    1723                 :        247 :   if (TREE_CODE (base) == MEM_REF)
    1724                 :            :     {
    1725                 :         44 :       off = build_int_cst (TREE_TYPE (TREE_OPERAND (base, 1)),
    1726                 :         44 :                            base_offset + offset / BITS_PER_UNIT);
    1727                 :         44 :       off = int_const_binop (PLUS_EXPR, TREE_OPERAND (base, 1), off);
    1728                 :         44 :       base = unshare_expr (TREE_OPERAND (base, 0));
    1729                 :            :     }
    1730                 :            :   else
    1731                 :            :     {
    1732                 :        203 :       off = build_int_cst (reference_alias_ptr_type (base),
    1733                 :        203 :                            base_offset + offset / BITS_PER_UNIT);
    1734                 :        203 :       base = build_fold_addr_expr (unshare_expr (base));
    1735                 :            :     }
    1736                 :            : 
    1737                 :        247 :   return fold_build2_loc (loc, MEM_REF, model->type, base, off);
    1738                 :            : }
    1739                 :            : 
    1740                 :            : /* Construct a memory reference consisting of component_refs and array_refs to
    1741                 :            :    a part of an aggregate *RES (which is of type TYPE).  The requested part
    1742                 :            :    should have type EXP_TYPE at be the given OFFSET.  This function might not
    1743                 :            :    succeed, it returns true when it does and only then *RES points to something
    1744                 :            :    meaningful.  This function should be used only to build expressions that we
    1745                 :            :    might need to present to user (e.g. in warnings).  In all other situations,
    1746                 :            :    build_ref_for_model or build_ref_for_offset should be used instead.  */
    1747                 :            : 
    1748                 :            : static bool
    1749                 :    2551420 : build_user_friendly_ref_for_offset (tree *res, tree type, HOST_WIDE_INT offset,
    1750                 :            :                                     tree exp_type)
    1751                 :            : {
    1752                 :    2572250 :   while (1)
    1753                 :            :     {
    1754                 :    2561830 :       tree fld;
    1755                 :    2561830 :       tree tr_size, index, minidx;
    1756                 :    2561830 :       HOST_WIDE_INT el_size;
    1757                 :            : 
    1758                 :    2561830 :       if (offset == 0 && exp_type
    1759                 :    2561830 :           && types_compatible_p (exp_type, type))
    1760                 :            :         return true;
    1761                 :            : 
    1762                 :    1511200 :       switch (TREE_CODE (type))
    1763                 :            :         {
    1764                 :    1498140 :         case UNION_TYPE:
    1765                 :    1498140 :         case QUAL_UNION_TYPE:
    1766                 :    1498140 :         case RECORD_TYPE:
    1767                 :    5484200 :           for (fld = TYPE_FIELDS (type); fld; fld = DECL_CHAIN (fld))
    1768                 :            :             {
    1769                 :    5475110 :               HOST_WIDE_INT pos, size;
    1770                 :    5475110 :               tree tr_pos, expr, *expr_ptr;
    1771                 :            : 
    1772                 :    5475110 :               if (TREE_CODE (fld) != FIELD_DECL)
    1773                 :    3976750 :                 continue;
    1774                 :            : 
    1775                 :    2479790 :               tr_pos = bit_position (fld);
    1776                 :    2479790 :               if (!tr_pos || !tree_fits_uhwi_p (tr_pos))
    1777                 :          0 :                 continue;
    1778                 :    2479790 :               pos = tree_to_uhwi (tr_pos);
    1779                 :    2479790 :               gcc_assert (TREE_CODE (type) == RECORD_TYPE || pos == 0);
    1780                 :    2479790 :               tr_size = DECL_SIZE (fld);
    1781                 :    2479790 :               if (!tr_size || !tree_fits_uhwi_p (tr_size))
    1782                 :          0 :                 continue;
    1783                 :    2479790 :               size = tree_to_uhwi (tr_size);
    1784                 :    2479790 :               if (size == 0)
    1785                 :            :                 {
    1786                 :       9309 :                   if (pos != offset)
    1787                 :       4659 :                     continue;
    1788                 :            :                 }
    1789                 :    2470480 :               else if (pos > offset || (pos + size) <= offset)
    1790                 :     976775 :                 continue;
    1791                 :            : 
    1792                 :    1498360 :               expr = build3 (COMPONENT_REF, TREE_TYPE (fld), *res, fld,
    1793                 :            :                              NULL_TREE);
    1794                 :    1498360 :               expr_ptr = &expr;
    1795                 :    1498360 :               if (build_user_friendly_ref_for_offset (expr_ptr, TREE_TYPE (fld),
    1796                 :            :                                                       offset - pos, exp_type))
    1797                 :            :                 {
    1798                 :    1490570 :                   *res = expr;
    1799                 :    1490570 :                   return true;
    1800                 :            :                 }
    1801                 :            :             }
    1802                 :            :           return false;
    1803                 :            : 
    1804                 :      10416 :         case ARRAY_TYPE:
    1805                 :      10416 :           tr_size = TYPE_SIZE (TREE_TYPE (type));
    1806                 :      10416 :           if (!tr_size || !tree_fits_uhwi_p (tr_size))
    1807                 :            :             return false;
    1808                 :      10416 :           el_size = tree_to_uhwi (tr_size);
    1809                 :            : 
    1810                 :      10416 :           minidx = TYPE_MIN_VALUE (TYPE_DOMAIN (type));
    1811                 :      10416 :           if (TREE_CODE (minidx) != INTEGER_CST || el_size == 0)
    1812                 :            :             return false;
    1813                 :      10416 :           index = build_int_cst (TYPE_DOMAIN (type), offset / el_size);
    1814                 :      10416 :           if (!integer_zerop (minidx))
    1815                 :        492 :             index = int_const_binop (PLUS_EXPR, index, minidx);
    1816                 :      10416 :           *res = build4 (ARRAY_REF, TREE_TYPE (type), *res, index,
    1817                 :            :                          NULL_TREE, NULL_TREE);
    1818                 :      10416 :           offset = offset % el_size;
    1819                 :      10416 :           type = TREE_TYPE (type);
    1820                 :      10416 :           break;
    1821                 :            : 
    1822                 :       2642 :         default:
    1823                 :       2642 :           if (offset != 0)
    1824                 :            :             return false;
    1825                 :            : 
    1826                 :       2546 :           if (exp_type)
    1827                 :            :             return false;
    1828                 :            :           else
    1829                 :          0 :             return true;
    1830                 :            :         }
    1831                 :      10416 :     }
    1832                 :            : }
    1833                 :            : 
    1834                 :            : /* Print message to dump file why a variable was rejected. */
    1835                 :            : 
    1836                 :            : static void
    1837                 :   10195600 : reject (tree var, const char *msg)
    1838                 :            : {
    1839                 :   10195600 :   if (dump_file && (dump_flags & TDF_DETAILS))
    1840                 :            :     {
    1841                 :         28 :       fprintf (dump_file, "Rejected (%d): %s: ", DECL_UID (var), msg);
    1842                 :         28 :       print_generic_expr (dump_file, var);
    1843                 :         28 :       fprintf (dump_file, "\n");
    1844                 :            :     }
    1845                 :   10195600 : }
    1846                 :            : 
    1847                 :            : /* Return true if VAR is a candidate for SRA.  */
    1848                 :            : 
    1849                 :            : static bool
    1850                 :   13256100 : maybe_add_sra_candidate (tree var)
    1851                 :            : {
    1852                 :   13256100 :   tree type = TREE_TYPE (var);
    1853                 :   13256100 :   const char *msg;
    1854                 :   13256100 :   tree_node **slot;
    1855                 :            : 
    1856                 :   13256100 :   if (!AGGREGATE_TYPE_P (type)) 
    1857                 :            :     {
    1858                 :    9016080 :       reject (var, "not aggregate");
    1859                 :    9016080 :       return false;
    1860                 :            :     }
    1861                 :            :   /* Allow constant-pool entries that "need to live in memory".  */
    1862                 :    4239990 :   if (needs_to_live_in_memory (var) && !constant_decl_p (var))
    1863                 :            :     {
    1864                 :    1178480 :       reject (var, "needs to live in memory");
    1865                 :    1178480 :       return false;
    1866                 :            :     }
    1867                 :    3061520 :   if (TREE_THIS_VOLATILE (var))
    1868                 :            :     {
    1869                 :        410 :       reject (var, "is volatile");
    1870                 :        410 :       return false;
    1871                 :            :     }
    1872                 :    3061110 :   if (!COMPLETE_TYPE_P (type))
    1873                 :            :     {
    1874                 :          0 :       reject (var, "has incomplete type");
    1875                 :          0 :       return false;
    1876                 :            :     }
    1877                 :    3061110 :   if (!tree_fits_uhwi_p (TYPE_SIZE (type)))
    1878                 :            :     {
    1879                 :         35 :       reject (var, "type size not fixed");
    1880                 :         35 :       return false;
    1881                 :            :     }
    1882                 :    3061070 :   if (tree_to_uhwi (TYPE_SIZE (type)) == 0)
    1883                 :            :     {
    1884                 :        389 :       reject (var, "type size is zero");
    1885                 :        389 :       return false;
    1886                 :            :     }
    1887                 :    3060680 :   if (type_internals_preclude_sra_p (type, &msg))
    1888                 :            :     {
    1889                 :        193 :       reject (var, msg);
    1890                 :        193 :       return false;
    1891                 :            :     }
    1892                 :    3060490 :   if (/* Fix for PR 41089.  tree-stdarg.c needs to have va_lists intact but
    1893                 :            :          we also want to schedule it rather late.  Thus we ignore it in
    1894                 :            :          the early pass. */
    1895                 :    3060490 :       (sra_mode == SRA_MODE_EARLY_INTRA
    1896                 :    3060490 :        && is_va_list_type (type)))
    1897                 :            :     {
    1898                 :          0 :       reject (var, "is va_list");
    1899                 :          0 :       return false;
    1900                 :            :     }
    1901                 :            : 
    1902                 :    3060490 :   bitmap_set_bit (candidate_bitmap, DECL_UID (var));
    1903                 :    3060490 :   slot = candidates->find_slot_with_hash (var, DECL_UID (var), INSERT);
    1904                 :    3060490 :   *slot = var;
    1905                 :            : 
    1906                 :    3060490 :   if (dump_file && (dump_flags & TDF_DETAILS))
    1907                 :            :     {
    1908                 :         14 :       fprintf (dump_file, "Candidate (%d): ", DECL_UID (var));
    1909                 :         14 :       print_generic_expr (dump_file, var);
    1910                 :         14 :       fprintf (dump_file, "\n");
    1911                 :            :     }
    1912                 :            : 
    1913                 :            :   return true;
    1914                 :            : }
    1915                 :            : 
    1916                 :            : /* The very first phase of intraprocedural SRA.  It marks in candidate_bitmap
    1917                 :            :    those with type which is suitable for scalarization.  */
    1918                 :            : 
    1919                 :            : static bool
    1920                 :    2266800 : find_var_candidates (void)
    1921                 :            : {
    1922                 :    2266800 :   tree var, parm;
    1923                 :    2266800 :   unsigned int i;
    1924                 :    2266800 :   bool ret = false;
    1925                 :            : 
    1926                 :    2266800 :   for (parm = DECL_ARGUMENTS (current_function_decl);
    1927                 :    7040860 :        parm;
    1928                 :    4774070 :        parm = DECL_CHAIN (parm))
    1929                 :    4774070 :     ret |= maybe_add_sra_candidate (parm);
    1930                 :            : 
    1931                 :   12711000 :   FOR_EACH_LOCAL_DECL (cfun, i, var)
    1932                 :            :     {
    1933                 :    8479950 :       if (!VAR_P (var))
    1934                 :          0 :         continue;
    1935                 :            : 
    1936                 :    8479950 :       ret |= maybe_add_sra_candidate (var);
    1937                 :            :     }
    1938                 :            : 
    1939                 :    2266800 :   return ret;
    1940                 :            : }
    1941                 :            : 
    1942                 :            : /* Return true if EXP is a reference chain of COMPONENT_REFs and AREAY_REFs
    1943                 :            :    ending either with a DECL or a MEM_REF with zero offset.  */
    1944                 :            : 
    1945                 :            : static bool
    1946                 :    6946760 : path_comparable_for_same_access (tree expr)
    1947                 :            : {
    1948                 :   11892100 :   while (handled_component_p (expr))
    1949                 :            :     {
    1950                 :    5023960 :       if (TREE_CODE (expr) == ARRAY_REF)
    1951                 :            :         {
    1952                 :            :           /* SSA name indices can occur here too when the array is of sie one.
    1953                 :            :              But we cannot just re-use array_refs with SSA names elsewhere in
    1954                 :            :              the function, so disallow non-constant indices.  TODO: Remove this
    1955                 :            :              limitation after teaching build_reconstructed_reference to replace
    1956                 :            :              the index with the index type lower bound.  */
    1957                 :     288117 :           if (TREE_CODE (TREE_OPERAND (expr, 1)) != INTEGER_CST)
    1958                 :            :             return false;
    1959                 :            :         }
    1960                 :    4945370 :       expr = TREE_OPERAND (expr, 0);
    1961                 :            :     }
    1962                 :            : 
    1963                 :    6868170 :   if (TREE_CODE (expr) == MEM_REF)
    1964                 :            :     {
    1965                 :     388099 :       if (!zerop (TREE_OPERAND (expr, 1)))
    1966                 :     152131 :         return false;
    1967                 :            :     }
    1968                 :            :   else
    1969                 :    6480070 :     gcc_assert (DECL_P (expr));
    1970                 :            : 
    1971                 :            :   return true;
    1972                 :            : }
    1973                 :            : 
    1974                 :            : /* Assuming that EXP1 consists of only COMPONENT_REFs and ARRAY_REFs, return
    1975                 :            :    true if the chain of these handled components are exactly the same as EXP2
    1976                 :            :    and the expression under them is the same DECL or an equivalent MEM_REF.
    1977                 :            :    The reference picked by compare_access_positions must go to EXP1.  */
    1978                 :            : 
    1979                 :            : static bool
    1980                 :    2660160 : same_access_path_p (tree exp1, tree exp2)
    1981                 :            : {
    1982                 :    2660160 :   if (TREE_CODE (exp1) != TREE_CODE (exp2))
    1983                 :            :     {
    1984                 :            :       /* Special case single-field structures loaded sometimes as the field
    1985                 :            :          and sometimes as the structure.  If the field is of a scalar type,
    1986                 :            :          compare_access_positions will put it into exp1.
    1987                 :            : 
    1988                 :            :          TODO: The gimple register type condition can be removed if teach
    1989                 :            :          compare_access_positions to put inner types first.  */
    1990                 :     229713 :       if (is_gimple_reg_type (TREE_TYPE (exp1))
    1991                 :     223448 :           && TREE_CODE (exp1) == COMPONENT_REF
    1992                 :     407859 :           && (TYPE_MAIN_VARIANT (TREE_TYPE (TREE_OPERAND (exp1, 0)))
    1993                 :     178146 :               == TYPE_MAIN_VARIANT (TREE_TYPE (exp2))))
    1994                 :     150680 :         exp1 = TREE_OPERAND (exp1, 0);
    1995                 :            :       else
    1996                 :            :         return false;
    1997                 :            :     }
    1998                 :            : 
    1999                 :    2581130 :   if (!operand_equal_p (exp1, exp2, OEP_ADDRESS_OF))
    2000                 :       9631 :     return false;
    2001                 :            : 
    2002                 :            :   return true;
    2003                 :            : }
    2004                 :            : 
    2005                 :            : /* Sort all accesses for the given variable, check for partial overlaps and
    2006                 :            :    return NULL if there are any.  If there are none, pick a representative for
    2007                 :            :    each combination of offset and size and create a linked list out of them.
    2008                 :            :    Return the pointer to the first representative and make sure it is the first
    2009                 :            :    one in the vector of accesses.  */
    2010                 :            : 
    2011                 :            : static struct access *
    2012                 :    2983060 : sort_and_splice_var_accesses (tree var)
    2013                 :            : {
    2014                 :    2983060 :   int i, j, access_count;
    2015                 :    2983060 :   struct access *res, **prev_acc_ptr = &res;
    2016                 :    2983060 :   vec<access_p> *access_vec;
    2017                 :    2983060 :   bool first = true;
    2018                 :    2983060 :   HOST_WIDE_INT low = -1, high = 0;
    2019                 :            : 
    2020                 :    2983060 :   access_vec = get_base_access_vector (var);
    2021                 :    2983060 :   if (!access_vec)
    2022                 :     125346 :     return NULL;
    2023                 :    2857710 :   access_count = access_vec->length ();
    2024                 :            : 
    2025                 :            :   /* Sort by <OFFSET, SIZE>.  */
    2026                 :    2857710 :   access_vec->qsort (compare_access_positions);
    2027                 :            : 
    2028                 :            :   i = 0;
    2029                 :    9282140 :   while (i < access_count)
    2030                 :            :     {
    2031                 :    6424900 :       struct access *access = (*access_vec)[i];
    2032                 :    6424900 :       bool grp_write = access->write;
    2033                 :    6424900 :       bool grp_read = !access->write;
    2034                 :    6424900 :       bool grp_scalar_write = access->write
    2035                 :    6424900 :         && is_gimple_reg_type (access->type);
    2036                 :    6424900 :       bool grp_scalar_read = !access->write
    2037                 :    6424900 :         && is_gimple_reg_type (access->type);
    2038                 :    6424900 :       bool grp_assignment_read = access->grp_assignment_read;
    2039                 :    6424900 :       bool grp_assignment_write = access->grp_assignment_write;
    2040                 :    6424900 :       bool multiple_scalar_reads = false;
    2041                 :    6424900 :       bool grp_partial_lhs = access->grp_partial_lhs;
    2042                 :    6424900 :       bool first_scalar = is_gimple_reg_type (access->type);
    2043                 :    6424900 :       bool unscalarizable_region = access->grp_unscalarizable_region;
    2044                 :    6424900 :       bool grp_same_access_path = true;
    2045                 :    6424900 :       bool bf_non_full_precision
    2046                 :    6424900 :         = (INTEGRAL_TYPE_P (access->type)
    2047                 :    1708040 :            && TYPE_PRECISION (access->type) != access->size
    2048                 :     110415 :            && TREE_CODE (access->expr) == COMPONENT_REF
    2049                 :    6466030 :            && DECL_BIT_FIELD (TREE_OPERAND (access->expr, 1)));
    2050                 :            : 
    2051                 :    6424900 :       if (first || access->offset >= high)
    2052                 :            :         {
    2053                 :    3098320 :           first = false;
    2054                 :    3098320 :           low = access->offset;
    2055                 :    3098320 :           high = access->offset + access->size;
    2056                 :            :         }
    2057                 :    3326570 :       else if (access->offset > low && access->offset + access->size > high)
    2058                 :            :         return NULL;
    2059                 :            :       else
    2060                 :    3326110 :         gcc_assert (access->offset >= low
    2061                 :            :                     && access->offset + access->size <= high);
    2062                 :            : 
    2063                 :    6424430 :       grp_same_access_path = path_comparable_for_same_access (access->expr);
    2064                 :            : 
    2065                 :    6424430 :       j = i + 1;
    2066                 :    9341550 :       while (j < access_count)
    2067                 :            :         {
    2068                 :    6484300 :           struct access *ac2 = (*access_vec)[j];
    2069                 :    6484300 :           if (ac2->offset != access->offset || ac2->size != access->size)
    2070                 :            :             break;
    2071                 :    2917120 :           if (ac2->write)
    2072                 :            :             {
    2073                 :     418986 :               grp_write = true;
    2074                 :     418986 :               grp_scalar_write = (grp_scalar_write
    2075                 :     418986 :                                   || is_gimple_reg_type (ac2->type));
    2076                 :            :             }
    2077                 :            :           else
    2078                 :            :             {
    2079                 :    2498130 :               grp_read = true;
    2080                 :    2498130 :               if (is_gimple_reg_type (ac2->type))
    2081                 :            :                 {
    2082                 :    1006520 :                   if (grp_scalar_read)
    2083                 :            :                     multiple_scalar_reads = true;
    2084                 :            :                   else
    2085                 :     197472 :                     grp_scalar_read = true;
    2086                 :            :                 }
    2087                 :            :             }
    2088                 :    2917120 :           grp_assignment_read |= ac2->grp_assignment_read;
    2089                 :    2917120 :           grp_assignment_write |= ac2->grp_assignment_write;
    2090                 :    2917120 :           grp_partial_lhs |= ac2->grp_partial_lhs;
    2091                 :    2917120 :           unscalarizable_region |= ac2->grp_unscalarizable_region;
    2092                 :    2917120 :           relink_to_new_repr (access, ac2);
    2093                 :            : 
    2094                 :            :           /* If there are both aggregate-type and scalar-type accesses with
    2095                 :            :              this combination of size and offset, the comparison function
    2096                 :            :              should have put the scalars first.  */
    2097                 :    2917120 :           gcc_assert (first_scalar || !is_gimple_reg_type (ac2->type));
    2098                 :            :           /* It also prefers integral types to non-integral.  However, when the
    2099                 :            :              precision of the selected type does not span the entire area and
    2100                 :            :              should also be used for a non-integer (i.e. float), we must not
    2101                 :            :              let that happen.  Normally analyze_access_subtree expands the type
    2102                 :            :              to cover the entire area but for bit-fields it doesn't.  */
    2103                 :    2917120 :           if (bf_non_full_precision && !INTEGRAL_TYPE_P (ac2->type))
    2104                 :            :             {
    2105                 :          0 :               if (dump_file && (dump_flags & TDF_DETAILS))
    2106                 :            :                 {
    2107                 :          0 :                   fprintf (dump_file, "Cannot scalarize the following access "
    2108                 :            :                            "because insufficient precision integer type was "
    2109                 :            :                            "selected.\n  ");
    2110                 :          0 :                   dump_access (dump_file, access, false);
    2111                 :            :                 }
    2112                 :            :               unscalarizable_region = true;
    2113                 :            :             }
    2114                 :            : 
    2115                 :    2917120 :           if (grp_same_access_path
    2116                 :    2917120 :               && !same_access_path_p (access->expr, ac2->expr))
    2117                 :            :             grp_same_access_path = false;
    2118                 :            : 
    2119                 :    2917120 :           ac2->group_representative = access;
    2120                 :    2917120 :           j++;
    2121                 :            :         }
    2122                 :            : 
    2123                 :    6424430 :       i = j;
    2124                 :            : 
    2125                 :    6424430 :       access->group_representative = access;
    2126                 :    6424430 :       access->grp_write = grp_write;
    2127                 :    6424430 :       access->grp_read = grp_read;
    2128                 :    6424430 :       access->grp_scalar_read = grp_scalar_read;
    2129                 :    6424430 :       access->grp_scalar_write = grp_scalar_write;
    2130                 :    6424430 :       access->grp_assignment_read = grp_assignment_read;
    2131                 :    6424430 :       access->grp_assignment_write = grp_assignment_write;
    2132                 :    6424430 :       access->grp_hint = multiple_scalar_reads && !constant_decl_p (var);
    2133                 :    6424430 :       access->grp_partial_lhs = grp_partial_lhs;
    2134                 :    6424430 :       access->grp_unscalarizable_region = unscalarizable_region;
    2135                 :    6424430 :       access->grp_same_access_path = grp_same_access_path;
    2136                 :            : 
    2137                 :    6424430 :       *prev_acc_ptr = access;
    2138                 :    6424430 :       prev_acc_ptr = &access->next_grp;
    2139                 :            :     }
    2140                 :            : 
    2141                 :    2857250 :   gcc_assert (res == (*access_vec)[0]);
    2142                 :            :   return res;
    2143                 :            : }
    2144                 :            : 
    2145                 :            : /* Create a variable for the given ACCESS which determines the type, name and a
    2146                 :            :    few other properties.  Return the variable declaration and store it also to
    2147                 :            :    ACCESS->replacement.  REG_TREE is used when creating a declaration to base a
    2148                 :            :    default-definition SSA name on in order to facilitate an uninitialized
    2149                 :            :    warning.  It is used instead of the actual ACCESS type if that is not of a
    2150                 :            :    gimple register type.  */
    2151                 :            : 
    2152                 :            : static tree
    2153                 :    3246440 : create_access_replacement (struct access *access, tree reg_type = NULL_TREE)
    2154                 :            : {
    2155                 :    3246440 :   tree repl;
    2156                 :            : 
    2157                 :    3246440 :   tree type = access->type;
    2158                 :    3246440 :   if (reg_type && !is_gimple_reg_type (type))
    2159                 :            :     type = reg_type;
    2160                 :            : 
    2161                 :    3246440 :   if (access->grp_to_be_debug_replaced)
    2162                 :            :     {
    2163                 :      73358 :       repl = create_tmp_var_raw (access->type);
    2164                 :      73358 :       DECL_CONTEXT (repl) = current_function_decl;
    2165                 :            :     }
    2166                 :            :   else
    2167                 :            :     /* Drop any special alignment on the type if it's not on the main
    2168                 :            :        variant.  This avoids issues with weirdo ABIs like AAPCS.  */
    2169                 :    3173080 :     repl = create_tmp_var (build_qualified_type (TYPE_MAIN_VARIANT (type),
    2170                 :    3173080 :                                                  TYPE_QUALS (type)), "SR");
    2171                 :    3246440 :   if (TREE_CODE (type) == COMPLEX_TYPE
    2172                 :    3246440 :       || TREE_CODE (type) == VECTOR_TYPE)
    2173                 :            :     {
    2174                 :       3011 :       if (!access->grp_partial_lhs)
    2175                 :       2520 :         DECL_GIMPLE_REG_P (repl) = 1;
    2176                 :            :     }
    2177                 :    3243430 :   else if (access->grp_partial_lhs
    2178                 :    3243430 :            && is_gimple_reg_type (type))
    2179                 :          0 :     TREE_ADDRESSABLE (repl) = 1;
    2180                 :            : 
    2181                 :    3246440 :   DECL_SOURCE_LOCATION (repl) = DECL_SOURCE_LOCATION (access->base);
    2182                 :    3246440 :   DECL_ARTIFICIAL (repl) = 1;
    2183                 :    3246440 :   DECL_IGNORED_P (repl) = DECL_IGNORED_P (access->base);
    2184                 :            : 
    2185                 :    3246440 :   if (DECL_NAME (access->base)
    2186                 :    2730520 :       && !DECL_IGNORED_P (access->base)
    2187                 :    3912600 :       && !DECL_ARTIFICIAL (access->base))
    2188                 :            :     {
    2189                 :     660545 :       char *pretty_name = make_fancy_name (access->expr);
    2190                 :     660545 :       tree debug_expr = unshare_expr_without_location (access->expr), d;
    2191                 :     660545 :       bool fail = false;
    2192                 :            : 
    2193                 :     660545 :       DECL_NAME (repl) = get_identifier (pretty_name);
    2194                 :     660545 :       DECL_NAMELESS (repl) = 1;
    2195                 :     660545 :       obstack_free (&name_obstack, pretty_name);
    2196                 :            : 
    2197                 :            :       /* Get rid of any SSA_NAMEs embedded in debug_expr,
    2198                 :            :          as DECL_DEBUG_EXPR isn't considered when looking for still
    2199                 :            :          used SSA_NAMEs and thus they could be freed.  All debug info
    2200                 :            :          generation cares is whether something is constant or variable
    2201                 :            :          and that get_ref_base_and_extent works properly on the
    2202                 :            :          expression.  It cannot handle accesses at a non-constant offset
    2203                 :            :          though, so just give up in those cases.  */
    2204                 :     818045 :       for (d = debug_expr;
    2205                 :    2212070 :            !fail && (handled_component_p (d) || TREE_CODE (d) == MEM_REF);
    2206                 :     818045 :            d = TREE_OPERAND (d, 0))
    2207                 :     818045 :         switch (TREE_CODE (d))
    2208                 :            :           {
    2209                 :      32545 :           case ARRAY_REF:
    2210                 :      32545 :           case ARRAY_RANGE_REF:
    2211                 :      32545 :             if (TREE_OPERAND (d, 1)
    2212                 :      32545 :                 && TREE_CODE (TREE_OPERAND (d, 1)) != INTEGER_CST)
    2213                 :            :               fail = true;
    2214                 :      32545 :             if (TREE_OPERAND (d, 3)
    2215                 :      32545 :                 && TREE_CODE (TREE_OPERAND (d, 3)) != INTEGER_CST)
    2216                 :            :               fail = true;
    2217                 :            :             /* FALLTHRU */
    2218                 :     745047 :           case COMPONENT_REF:
    2219                 :     745047 :             if (TREE_OPERAND (d, 2)
    2220                 :     745047 :                 && TREE_CODE (TREE_OPERAND (d, 2)) != INTEGER_CST)
    2221                 :            :               fail = true;
    2222                 :            :             break;
    2223                 :      72998 :           case MEM_REF:
    2224                 :      72998 :             if (TREE_CODE (TREE_OPERAND (d, 0)) != ADDR_EXPR)
    2225                 :            :               fail = true;
    2226                 :            :             else
    2227                 :      72998 :               d = TREE_OPERAND (d, 0);
    2228                 :            :             break;
    2229                 :            :           default:
    2230                 :            :             break;
    2231                 :            :           }
    2232                 :     660545 :       if (!fail)
    2233                 :            :         {
    2234                 :     660485 :           SET_DECL_DEBUG_EXPR (repl, debug_expr);
    2235                 :     660485 :           DECL_HAS_DEBUG_EXPR_P (repl) = 1;
    2236                 :            :         }
    2237                 :     660545 :       if (access->grp_no_warning)
    2238                 :        278 :         TREE_NO_WARNING (repl) = 1;
    2239                 :            :       else
    2240                 :     660267 :         TREE_NO_WARNING (repl) = TREE_NO_WARNING (access->base);
    2241                 :            :     }
    2242                 :            :   else
    2243                 :    2585900 :     TREE_NO_WARNING (repl) = 1;
    2244                 :            : 
    2245                 :    3246440 :   if (dump_file)
    2246                 :            :     {
    2247                 :         67 :       if (access->grp_to_be_debug_replaced)
    2248                 :            :         {
    2249                 :          4 :           fprintf (dump_file, "Created a debug-only replacement for ");
    2250                 :          4 :           print_generic_expr (dump_file, access->base);
    2251                 :          4 :           fprintf (dump_file, " offset: %u, size: %u\n",
    2252                 :          4 :                    (unsigned) access->offset, (unsigned) access->size);
    2253                 :            :         }
    2254                 :            :       else
    2255                 :            :         {
    2256                 :         63 :           fprintf (dump_file, "Created a replacement for ");
    2257                 :         63 :           print_generic_expr (dump_file, access->base);
    2258                 :         63 :           fprintf (dump_file, " offset: %u, size: %u: ",
    2259                 :         63 :                    (unsigned) access->offset, (unsigned) access->size);
    2260                 :         63 :           print_generic_expr (dump_file, repl);
    2261                 :         63 :           fprintf (dump_file, "\n");
    2262                 :            :         }
    2263                 :            :     }
    2264                 :    3246440 :   sra_stats.replacements++;
    2265                 :            : 
    2266                 :    3246440 :   return repl;
    2267                 :            : }
    2268                 :            : 
    2269                 :            : /* Return ACCESS scalar replacement, which must exist.  */
    2270                 :            : 
    2271                 :            : static inline tree
    2272                 :    9600080 : get_access_replacement (struct access *access)
    2273                 :            : {
    2274                 :      25880 :   gcc_checking_assert (access->replacement_decl);
    2275                 :    9575390 :   return access->replacement_decl;
    2276                 :            : }
    2277                 :            : 
    2278                 :            : 
    2279                 :            : /* Build a subtree of accesses rooted in *ACCESS, and move the pointer in the
    2280                 :            :    linked list along the way.  Stop when *ACCESS is NULL or the access pointed
    2281                 :            :    to it is not "within" the root.  Return false iff some accesses partially
    2282                 :            :    overlap.  */
    2283                 :            : 
    2284                 :            : static bool
    2285                 :    6420140 : build_access_subtree (struct access **access)
    2286                 :            : {
    2287                 :    6420140 :   struct access *root = *access, *last_child = NULL;
    2288                 :    6420140 :   HOST_WIDE_INT limit = root->offset + root->size;
    2289                 :            : 
    2290                 :    6420140 :   *access = (*access)->next_grp;
    2291                 :    9741560 :   while  (*access && (*access)->offset + (*access)->size <= limit)
    2292                 :            :     {
    2293                 :    3322310 :       if (!last_child)
    2294                 :    1444730 :         root->first_child = *access;
    2295                 :            :       else
    2296                 :    1877590 :         last_child->next_sibling = *access;
    2297                 :    3322310 :       last_child = *access;
    2298                 :    3322310 :       (*access)->parent = root;
    2299                 :    3322310 :       (*access)->grp_write |= root->grp_write;
    2300                 :            : 
    2301                 :    3322310 :       if (!build_access_subtree (access))
    2302                 :            :         return false;
    2303                 :            :     }
    2304                 :            : 
    2305                 :    6419240 :   if (*access && (*access)->offset < limit)
    2306                 :        792 :     return false;
    2307                 :            : 
    2308                 :            :   return true;
    2309                 :            : }
    2310                 :            : 
    2311                 :            : /* Build a tree of access representatives, ACCESS is the pointer to the first
    2312                 :            :    one, others are linked in a list by the next_grp field.  Return false iff
    2313                 :            :    some accesses partially overlap.  */
    2314                 :            : 
    2315                 :            : static bool
    2316                 :    2857250 : build_access_trees (struct access *access)
    2317                 :            : {
    2318                 :    5954290 :   while (access)
    2319                 :            :     {
    2320                 :    3097830 :       struct access *root = access;
    2321                 :            : 
    2322                 :    3097830 :       if (!build_access_subtree (&access))
    2323                 :            :         return false;
    2324                 :    3097040 :       root->next_grp = access;
    2325                 :            :     }
    2326                 :            :   return true;
    2327                 :            : }
    2328                 :            : 
    2329                 :            : /* Traverse the access forest where ROOT is the first root and verify that
    2330                 :            :    various important invariants hold true.  */
    2331                 :            : 
    2332                 :            : DEBUG_FUNCTION void
    2333                 :    2856460 : verify_sra_access_forest (struct access *root)
    2334                 :            : {
    2335                 :    2856460 :   struct access *access = root;
    2336                 :    2856460 :   tree first_base = root->base;
    2337                 :    2856460 :   gcc_assert (DECL_P (first_base));
    2338                 :    7906340 :   do
    2339                 :            :     {
    2340                 :    7906340 :       gcc_assert (access->base == first_base);
    2341                 :    7906340 :       if (access->parent)
    2342                 :    4809300 :         gcc_assert (access->offset >= access->parent->offset
    2343                 :            :                     && access->size <= access->parent->size);
    2344                 :    7906340 :       if (access->next_sibling)
    2345                 :    2687420 :         gcc_assert (access->next_sibling->offset
    2346                 :            :                     >= access->offset + access->size);
    2347                 :            : 
    2348                 :    7906340 :       poly_int64 poffset, psize, pmax_size;
    2349                 :    7906340 :       bool reverse;
    2350                 :    7906340 :       tree base = get_ref_base_and_extent (access->expr, &poffset, &psize,
    2351                 :            :                                            &pmax_size, &reverse);
    2352                 :    7906340 :       HOST_WIDE_INT offset, size, max_size;
    2353                 :    7906340 :       if (!poffset.is_constant (&offset)
    2354                 :    7906340 :           || !psize.is_constant (&size)
    2355                 :    7906340 :           || !pmax_size.is_constant (&max_size))
    2356                 :            :         gcc_unreachable ();
    2357                 :    7906340 :       gcc_assert (base == first_base);
    2358                 :    7906340 :       gcc_assert (offset == access->offset);
    2359                 :    7906340 :       gcc_assert (access->grp_unscalarizable_region
    2360                 :            :                   || size == max_size);
    2361                 :    7906340 :       gcc_assert (!is_gimple_reg_type (access->type)
    2362                 :            :                   || max_size == access->size);
    2363                 :    7906340 :       gcc_assert (reverse == access->reverse);
    2364                 :            : 
    2365                 :    7906340 :       if (access->first_child)
    2366                 :            :         {
    2367                 :    2121880 :           gcc_assert (access->first_child->parent == access);
    2368                 :            :           access = access->first_child;
    2369                 :            :         }
    2370                 :    5784460 :       else if (access->next_sibling)
    2371                 :            :         {
    2372                 :    2539490 :           gcc_assert (access->next_sibling->parent == access->parent);
    2373                 :            :           access = access->next_sibling;
    2374                 :            :         }
    2375                 :            :       else
    2376                 :            :         {
    2377                 :    5366850 :           while (access->parent && !access->next_sibling)
    2378                 :            :             access = access->parent;
    2379                 :    3244970 :           if (access->next_sibling)
    2380                 :            :             access = access->next_sibling;
    2381                 :            :           else
    2382                 :            :             {
    2383                 :    3097040 :               gcc_assert (access == root);
    2384                 :    3097040 :               root = root->next_grp;
    2385                 :    3097040 :               access = root;
    2386                 :            :             }
    2387                 :            :         }
    2388                 :            :     }
    2389                 :    7906340 :   while (access);
    2390                 :    2856460 : }
    2391                 :            : 
    2392                 :            : /* Verify access forests of all candidates with accesses by calling
    2393                 :            :    verify_access_forest on each on them.  */
    2394                 :            : 
    2395                 :            : DEBUG_FUNCTION void
    2396                 :     448277 : verify_all_sra_access_forests (void)
    2397                 :            : {
    2398                 :     448277 :   bitmap_iterator bi;
    2399                 :     448277 :   unsigned i;
    2400                 :    3304730 :   EXECUTE_IF_SET_IN_BITMAP (candidate_bitmap, 0, i, bi)
    2401                 :            :     {
    2402                 :    2856460 :       tree var = candidate (i);
    2403                 :    2856460 :       struct access *access = get_first_repr_for_decl (var);
    2404                 :    2856460 :       if (access)
    2405                 :            :         {
    2406                 :    2856460 :           gcc_assert (access->base == var);
    2407                 :    2856460 :           verify_sra_access_forest (access);
    2408                 :            :         }
    2409                 :            :     }
    2410                 :     448277 : }
    2411                 :            : 
    2412                 :            : /* Return true if expr contains some ARRAY_REFs into a variable bounded
    2413                 :            :    array.  */
    2414                 :            : 
    2415                 :            : static bool
    2416                 :    7748390 : expr_with_var_bounded_array_refs_p (tree expr)
    2417                 :            : {
    2418                 :   14080600 :   while (handled_component_p (expr))
    2419                 :            :     {
    2420                 :    6332200 :       if (TREE_CODE (expr) == ARRAY_REF
    2421                 :    6332200 :           && !tree_fits_shwi_p (array_ref_low_bound (expr)))
    2422                 :            :         return true;
    2423                 :    6332200 :       expr = TREE_OPERAND (expr, 0);
    2424                 :            :     }
    2425                 :            :   return false;
    2426                 :            : }
    2427                 :            : 
    2428                 :            : /* Analyze the subtree of accesses rooted in ROOT, scheduling replacements when
    2429                 :            :    both seeming beneficial and when ALLOW_REPLACEMENTS allows it.  If TOTALLY
    2430                 :            :    is set, we are totally scalarizing the aggregate.  Also set all sorts of
    2431                 :            :    access flags appropriately along the way, notably always set grp_read and
    2432                 :            :    grp_assign_read according to MARK_READ and grp_write when MARK_WRITE is
    2433                 :            :    true.
    2434                 :            : 
    2435                 :            :    Creating a replacement for a scalar access is considered beneficial if its
    2436                 :            :    grp_hint ot TOTALLY is set (this means either that there is more than one
    2437                 :            :    direct read access or that we are attempting total scalarization) or
    2438                 :            :    according to the following table:
    2439                 :            : 
    2440                 :            :    Access written to through a scalar type (once or more times)
    2441                 :            :    |
    2442                 :            :    |    Written to in an assignment statement
    2443                 :            :    |    |
    2444                 :            :    |    |       Access read as scalar _once_
    2445                 :            :    |    |       |
    2446                 :            :    |    |       |       Read in an assignment statement
    2447                 :            :    |    |       |       |
    2448                 :            :    |    |       |       |       Scalarize       Comment
    2449                 :            : -----------------------------------------------------------------------------
    2450                 :            :    0    0       0       0                       No access for the scalar
    2451                 :            :    0    0       0       1                       No access for the scalar
    2452                 :            :    0    0       1       0       No              Single read - won't help
    2453                 :            :    0    0       1       1       No              The same case
    2454                 :            :    0    1       0       0                       No access for the scalar
    2455                 :            :    0    1       0       1                       No access for the scalar
    2456                 :            :    0    1       1       0       Yes             s = *g; return s.i;
    2457                 :            :    0    1       1       1       Yes             The same case as above
    2458                 :            :    1    0       0       0       No              Won't help
    2459                 :            :    1    0       0       1       Yes             s.i = 1; *g = s;
    2460                 :            :    1    0       1       0       Yes             s.i = 5; g = s.i;
    2461                 :            :    1    0       1       1       Yes             The same case as above
    2462                 :            :    1    1       0       0       No              Won't help.
    2463                 :            :    1    1       0       1       Yes             s.i = 1; *g = s;
    2464                 :            :    1    1       1       0       Yes             s = *g; return s.i;
    2465                 :            :    1    1       1       1       Yes             Any of the above yeses  */
    2466                 :            : 
    2467                 :            : static bool
    2468                 :    7906340 : analyze_access_subtree (struct access *root, struct access *parent,
    2469                 :            :                         bool allow_replacements, bool totally)
    2470                 :            : {
    2471                 :    7906340 :   struct access *child;
    2472                 :    7906340 :   HOST_WIDE_INT limit = root->offset + root->size;
    2473                 :    7906340 :   HOST_WIDE_INT covered_to = root->offset;
    2474                 :    7906340 :   bool scalar = is_gimple_reg_type (root->type);
    2475                 :    7906340 :   bool hole = false, sth_created = false;
    2476                 :            : 
    2477                 :    7906340 :   if (parent)
    2478                 :            :     {
    2479                 :    4809300 :       if (parent->grp_read)
    2480                 :    3936480 :         root->grp_read = 1;
    2481                 :    4809300 :       if (parent->grp_assignment_read)
    2482                 :    2460570 :         root->grp_assignment_read = 1;
    2483                 :    4809300 :       if (parent->grp_write)
    2484                 :    2339990 :         root->grp_write = 1;
    2485                 :    4809300 :       if (parent->grp_assignment_write)
    2486                 :    2437100 :         root->grp_assignment_write = 1;
    2487                 :    4809300 :       if (!parent->grp_same_access_path)
    2488                 :     150778 :         root->grp_same_access_path = 0;
    2489                 :            :     }
    2490                 :            : 
    2491                 :    7906340 :   if (root->grp_unscalarizable_region)
    2492                 :            :     allow_replacements = false;
    2493                 :            : 
    2494                 :    7825470 :   if (allow_replacements && expr_with_var_bounded_array_refs_p (root->expr))
    2495                 :            :     allow_replacements = false;
    2496                 :            : 
    2497                 :   12715600 :   for (child = root->first_child; child; child = child->next_sibling)
    2498                 :            :     {
    2499                 :    4809300 :       hole |= covered_to < child->offset;
    2500                 :    9618610 :       sth_created |= analyze_access_subtree (child, root,
    2501                 :    4809300 :                                              allow_replacements && !scalar,
    2502                 :            :                                              totally);
    2503                 :            : 
    2504                 :    4809300 :       root->grp_unscalarized_data |= child->grp_unscalarized_data;
    2505                 :    4809300 :       if (child->grp_covered)
    2506                 :    3111600 :         covered_to += child->size;
    2507                 :            :       else
    2508                 :            :         hole = true;
    2509                 :            :     }
    2510                 :            : 
    2511                 :    7906340 :   if (allow_replacements && scalar && !root->first_child
    2512                 :    5020490 :       && (totally || !root->grp_total_scalarization)
    2513                 :    5020490 :       && (totally
    2514                 :    2955410 :           || root->grp_hint
    2515                 :    2556550 :           || ((root->grp_scalar_read || root->grp_assignment_read)
    2516                 :    1079450 :               && (root->grp_scalar_write || root->grp_assignment_write))))
    2517                 :            :     {
    2518                 :            :       /* Always create access replacements that cover the whole access.
    2519                 :            :          For integral types this means the precision has to match.
    2520                 :            :          Avoid assumptions based on the integral type kind, too.  */
    2521                 :    3172650 :       if (INTEGRAL_TYPE_P (root->type)
    2522                 :    1206090 :           && (TREE_CODE (root->type) != INTEGER_TYPE
    2523                 :    1157470 :               || TYPE_PRECISION (root->type) != root->size)
    2524                 :            :           /* But leave bitfield accesses alone.  */
    2525                 :    3221270 :           && (TREE_CODE (root->expr) != COMPONENT_REF
    2526                 :      47947 :               || !DECL_BIT_FIELD (TREE_OPERAND (root->expr, 1))))
    2527                 :            :         {
    2528                 :      48445 :           tree rt = root->type;
    2529                 :      48445 :           gcc_assert ((root->offset % BITS_PER_UNIT) == 0
    2530                 :            :                       && (root->size % BITS_PER_UNIT) == 0);
    2531                 :      48445 :           root->type = build_nonstandard_integer_type (root->size,
    2532                 :      48445 :                                                        TYPE_UNSIGNED (rt));
    2533                 :      96890 :           root->expr = build_ref_for_offset (UNKNOWN_LOCATION, root->base,
    2534                 :      48445 :                                              root->offset, root->reverse,
    2535                 :      48445 :                                              root->type, NULL, false);
    2536                 :            : 
    2537                 :      48445 :           if (dump_file && (dump_flags & TDF_DETAILS))
    2538                 :            :             {
    2539                 :          0 :               fprintf (dump_file, "Changing the type of a replacement for ");
    2540                 :          0 :               print_generic_expr (dump_file, root->base);
    2541                 :          0 :               fprintf (dump_file, " offset: %u, size: %u ",
    2542                 :          0 :                        (unsigned) root->offset, (unsigned) root->size);
    2543                 :          0 :               fprintf (dump_file, " to an integer.\n");
    2544                 :            :             }
    2545                 :            :         }
    2546                 :            : 
    2547                 :    3172650 :       root->grp_to_be_replaced = 1;
    2548                 :    3172650 :       root->replacement_decl = create_access_replacement (root);
    2549                 :    3172650 :       sth_created = true;
    2550                 :    3172650 :       hole = false;
    2551                 :            :     }
    2552                 :            :   else
    2553                 :            :     {
    2554                 :    4733690 :       if (allow_replacements
    2555                 :    1851280 :           && scalar && !root->first_child
    2556                 :    1847840 :           && !root->grp_total_scalarization
    2557                 :    1847840 :           && (root->grp_scalar_write || root->grp_assignment_write)
    2558                 :    6210790 :           && !bitmap_bit_p (cannot_scalarize_away_bitmap,
    2559                 :    1477100 :                             DECL_UID (root->base)))
    2560                 :            :         {
    2561                 :     200005 :           gcc_checking_assert (!root->grp_scalar_read
    2562                 :            :                                && !root->grp_assignment_read);
    2563                 :     200005 :           sth_created = true;
    2564                 :     200005 :           if (MAY_HAVE_DEBUG_BIND_STMTS)
    2565                 :            :             {
    2566                 :      73358 :               root->grp_to_be_debug_replaced = 1;
    2567                 :      73358 :               root->replacement_decl = create_access_replacement (root);
    2568                 :            :             }
    2569                 :            :         }
    2570                 :            : 
    2571                 :    4733690 :       if (covered_to < limit)
    2572                 :    3512420 :         hole = true;
    2573                 :    4733690 :       if (scalar || !allow_replacements)
    2574                 :    2009230 :         root->grp_total_scalarization = 0;
    2575                 :            :     }
    2576                 :            : 
    2577                 :    7906340 :   if (!hole || totally)
    2578                 :    4473280 :     root->grp_covered = 1;
    2579                 :    3433060 :   else if (root->grp_write || comes_initialized_p (root->base))
    2580                 :    2847870 :     root->grp_unscalarized_data = 1; /* not covered and written to */
    2581                 :    7906340 :   return sth_created;
    2582                 :            : }
    2583                 :            : 
    2584                 :            : /* Analyze all access trees linked by next_grp by the means of
    2585                 :            :    analyze_access_subtree.  */
    2586                 :            : static bool
    2587                 :    2856460 : analyze_access_trees (struct access *access)
    2588                 :            : {
    2589                 :    2856460 :   bool ret = false;
    2590                 :            : 
    2591                 :    5953490 :   while (access)
    2592                 :            :     {
    2593                 :    3097040 :       if (analyze_access_subtree (access, NULL, true,
    2594                 :    3097040 :                                   access->grp_total_scalarization))
    2595                 :    1600510 :         ret = true;
    2596                 :    3097040 :       access = access->next_grp;
    2597                 :            :     }
    2598                 :            : 
    2599                 :    2856460 :   return ret;
    2600                 :            : }
    2601                 :            : 
    2602                 :            : /* Return true iff a potential new child of ACC at offset OFFSET and with size
    2603                 :            :    SIZE would conflict with an already existing one.  If exactly such a child
    2604                 :            :    already exists in ACC, store a pointer to it in EXACT_MATCH.  */
    2605                 :            : 
    2606                 :            : static bool
    2607                 :    4907160 : child_would_conflict_in_acc (struct access *acc, HOST_WIDE_INT norm_offset,
    2608                 :            :                               HOST_WIDE_INT size, struct access **exact_match)
    2609                 :            : {
    2610                 :    4907160 :   struct access *child;
    2611                 :            : 
    2612                 :    8350030 :   for (child = acc->first_child; child; child = child->next_sibling)
    2613                 :            :     {
    2614                 :    7377160 :       if (child->offset == norm_offset && child->size == size)
    2615                 :            :         {
    2616                 :          0 :           *exact_match = child;
    2617                 :          0 :           return true;
    2618                 :            :         }
    2619                 :            : 
    2620                 :    3447400 :       if (child->offset < norm_offset + size
    2621                 :    3420260 :           && child->offset + child->size > norm_offset)
    2622                 :            :         return true;
    2623                 :            :     }
    2624                 :            : 
    2625                 :            :   return false;
    2626                 :            : }
    2627                 :            : 
    2628                 :            : /* Create a new child access of PARENT, with all properties just like MODEL
    2629                 :            :    except for its offset and with its grp_write false and grp_read true.
    2630                 :            :    Return the new access or NULL if it cannot be created.  Note that this
    2631                 :            :    access is created long after all splicing and sorting, it's not located in
    2632                 :            :    any access vector and is automatically a representative of its group.  Set
    2633                 :            :    the gpr_write flag of the new accesss if SET_GRP_WRITE is true.  */
    2634                 :            : 
    2635                 :            : static struct access *
    2636                 :     966052 : create_artificial_child_access (struct access *parent, struct access *model,
    2637                 :            :                                 HOST_WIDE_INT new_offset,
    2638                 :            :                                 bool set_grp_read, bool set_grp_write)
    2639                 :            : {
    2640                 :     966052 :   struct access **child;
    2641                 :     966052 :   tree expr = parent->base;
    2642                 :            : 
    2643                 :     966052 :   gcc_assert (!model->grp_unscalarizable_region);
    2644                 :            : 
    2645                 :     966052 :   struct access *access = access_pool.allocate ();
    2646                 :     966052 :   memset (access, 0, sizeof (struct access));
    2647                 :     966052 :   if (!build_user_friendly_ref_for_offset (&expr, TREE_TYPE (expr), new_offset,
    2648                 :            :                                            model->type))
    2649                 :            :     {
    2650                 :       2368 :       access->grp_no_warning = true;
    2651                 :       2368 :       expr = build_ref_for_model (EXPR_LOCATION (parent->base), parent->base,
    2652                 :            :                                   new_offset, model, NULL, false);
    2653                 :            :     }
    2654                 :            : 
    2655                 :     966052 :   access->base = parent->base;
    2656                 :     966052 :   access->expr = expr;
    2657                 :     966052 :   access->offset = new_offset;
    2658                 :     966052 :   access->size = model->size;
    2659                 :     966052 :   access->type = model->type;
    2660                 :     966052 :   access->parent = parent;
    2661                 :     966052 :   access->grp_read = set_grp_read;
    2662                 :     966052 :   access->grp_write = set_grp_write;
    2663                 :     966052 :   access->reverse = model->reverse;
    2664                 :            : 
    2665                 :     966052 :   child = &parent->first_child;
    2666                 :    1617790 :   while (*child && (*child)->offset < new_offset)
    2667                 :     651739 :     child = &(*child)->next_sibling;
    2668                 :            : 
    2669                 :     966052 :   access->next_sibling = *child;
    2670                 :     966052 :   *child = access;
    2671                 :            : 
    2672                 :     966052 :   return access;
    2673                 :            : }
    2674                 :            : 
    2675                 :            : 
    2676                 :            : /* Beginning with ACCESS, traverse its whole access subtree and mark all
    2677                 :            :    sub-trees as written to.  If any of them has not been marked so previously
    2678                 :            :    and has assignment links leading from it, re-enqueue it.  */
    2679                 :            : 
    2680                 :            : static void
    2681                 :     668302 : subtree_mark_written_and_rhs_enqueue (struct access *access)
    2682                 :            : {
    2683                 :     668302 :   if (access->grp_write)
    2684                 :            :     return;
    2685                 :     647505 :   access->grp_write = true;
    2686                 :     647505 :   add_access_to_rhs_work_queue (access);
    2687                 :            : 
    2688                 :     647505 :   struct access *child;
    2689                 :     806586 :   for (child = access->first_child; child; child = child->next_sibling)
    2690                 :     159081 :     subtree_mark_written_and_rhs_enqueue (child);
    2691                 :            : }
    2692                 :            : 
    2693                 :            : /* If there is still budget to create a propagation access for DECL, return
    2694                 :            :    true and decrement the budget.  Otherwise return false.  */
    2695                 :            : 
    2696                 :            : static bool
    2697                 :     972490 : budget_for_propagation_access (tree decl)
    2698                 :            : {
    2699                 :     972490 :   unsigned b, *p = propagation_budget->get (decl);
    2700                 :     562433 :   if (p)
    2701                 :     562433 :     b = *p;
    2702                 :            :   else
    2703                 :     410057 :     b = param_sra_max_propagations;
    2704                 :            : 
    2705                 :     972490 :   if (b == 0)
    2706                 :            :     return false;
    2707                 :     966056 :   b--;
    2708                 :            : 
    2709                 :     966056 :   if (b == 0 && dump_file && (dump_flags & TDF_DETAILS))
    2710                 :            :     {
    2711                 :          0 :       fprintf (dump_file, "The propagation budget of ");
    2712                 :          0 :       print_generic_expr (dump_file, decl);
    2713                 :          0 :       fprintf (dump_file, " (UID: %u) has been exhausted.\n", DECL_UID (decl));
    2714                 :            :     }
    2715                 :     966056 :   propagation_budget->put (decl, b);
    2716                 :     966056 :   return true;
    2717                 :            : }
    2718                 :            : 
    2719                 :            : /* Propagate subaccesses and grp_write flags of RACC across an assignment link
    2720                 :            :    to LACC.  Enqueue sub-accesses as necessary so that the write flag is
    2721                 :            :    propagated transitively.  Return true if anything changed.  Additionally, if
    2722                 :            :    RACC is a scalar access but LACC is not, change the type of the latter, if
    2723                 :            :    possible.  */
    2724                 :            : 
    2725                 :            : static bool
    2726                 :    2673140 : propagate_subaccesses_from_rhs (struct access *lacc, struct access *racc)
    2727                 :            : {
    2728                 :    2673140 :   struct access *rchild;
    2729                 :    2673140 :   HOST_WIDE_INT norm_delta = lacc->offset - racc->offset;
    2730                 :    2673140 :   bool ret = false;
    2731                 :            : 
    2732                 :            :   /* IF the LHS is still not marked as being written to, we only need to do so
    2733                 :            :      if the RHS at this level actually was.  */
    2734                 :    2673140 :   if (!lacc->grp_write)
    2735                 :            :     {
    2736                 :     859133 :       gcc_checking_assert (!comes_initialized_p (racc->base));
    2737                 :     859133 :       if (racc->grp_write)
    2738                 :            :         {
    2739                 :     424632 :           subtree_mark_written_and_rhs_enqueue (lacc);
    2740                 :     424632 :           ret = true;
    2741                 :            :         }
    2742                 :            :     }
    2743                 :            : 
    2744                 :    2673140 :   if (is_gimple_reg_type (lacc->type)
    2745                 :    1857580 :       || lacc->grp_unscalarizable_region
    2746                 :    4529910 :       || racc->grp_unscalarizable_region)
    2747                 :            :     {
    2748                 :     816752 :       if (!lacc->grp_write)
    2749                 :            :         {
    2750                 :      35172 :           ret = true;
    2751                 :      35172 :           subtree_mark_written_and_rhs_enqueue (lacc);
    2752                 :            :         }
    2753                 :     816752 :       return ret;
    2754                 :            :     }
    2755                 :            : 
    2756                 :    1856390 :   if (is_gimple_reg_type (racc->type))
    2757                 :            :     {
    2758                 :      87235 :       if (!lacc->grp_write)
    2759                 :            :         {
    2760                 :        706 :           ret = true;
    2761                 :        706 :           subtree_mark_written_and_rhs_enqueue (lacc);
    2762                 :            :         }
    2763                 :      87235 :       if (!lacc->first_child && !racc->first_child)
    2764                 :            :         {
    2765                 :      87006 :           tree t = lacc->base;
    2766                 :            : 
    2767                 :      87006 :           lacc->type = racc->type;
    2768                 :      87006 :           if (build_user_friendly_ref_for_offset (&t, TREE_TYPE (t),
    2769                 :            :                                                   lacc->offset, racc->type))
    2770                 :            :             {
    2771                 :      86953 :               lacc->expr = t;
    2772                 :      86953 :               lacc->grp_same_access_path = true;
    2773                 :            :             }
    2774                 :            :           else
    2775                 :            :             {
    2776                 :         53 :               lacc->expr = build_ref_for_model (EXPR_LOCATION (lacc->base),
    2777                 :            :                                                 lacc->base, lacc->offset,
    2778                 :            :                                                 racc, NULL, false);
    2779                 :         53 :               lacc->grp_no_warning = true;
    2780                 :         53 :               lacc->grp_same_access_path = false;
    2781                 :            :             }
    2782                 :            :         }
    2783                 :      87235 :       return ret;
    2784                 :            :     }
    2785                 :            : 
    2786                 :    4137190 :   for (rchild = racc->first_child; rchild; rchild = rchild->next_sibling)
    2787                 :            :     {
    2788                 :    2368040 :       struct access *new_acc = NULL;
    2789                 :    2368040 :       HOST_WIDE_INT norm_offset = rchild->offset + norm_delta;
    2790                 :            : 
    2791                 :    2968260 :       if (child_would_conflict_in_acc (lacc, norm_offset, rchild->size,
    2792                 :            :                                         &new_acc))
    2793                 :            :         {
    2794                 :    1770370 :           if (new_acc)
    2795                 :            :             {
    2796                 :    1767820 :               if (!new_acc->grp_write && rchild->grp_write)
    2797                 :            :                 {
    2798                 :      47513 :                   gcc_assert (!lacc->grp_write);
    2799                 :      47513 :                   subtree_mark_written_and_rhs_enqueue (new_acc);
    2800                 :      47513 :                   ret = true;
    2801                 :            :                 }
    2802                 :            : 
    2803                 :    1767820 :               rchild->grp_hint = 1;
    2804                 :    1767820 :               new_acc->grp_hint |= new_acc->grp_read;
    2805                 :    1767820 :               if (rchild->first_child
    2806                 :    1767820 :                   && propagate_subaccesses_from_rhs (new_acc, rchild))
    2807                 :            :                 {
    2808                 :       2507 :                   ret = 1;
    2809                 :       2507 :                   add_access_to_rhs_work_queue (new_acc);
    2810                 :            :                 }
    2811                 :            :             }
    2812                 :            :           else
    2813                 :            :             {
    2814                 :       2549 :               if (!lacc->grp_write)
    2815                 :            :                 {
    2816                 :        558 :                   ret = true;
    2817                 :        558 :                   subtree_mark_written_and_rhs_enqueue (lacc);
    2818                 :            :                 }
    2819                 :            :             }
    2820                 :    2368040 :           continue;
    2821                 :            :         }
    2822                 :            : 
    2823                 :     604471 :       if (rchild->grp_unscalarizable_region
    2824                 :     597668 :           || !budget_for_propagation_access (lacc->base))
    2825                 :            :         {
    2826                 :       6803 :           if (rchild->grp_write && !lacc->grp_write)
    2827                 :            :             {
    2828                 :        583 :               ret = true;
    2829                 :        583 :               subtree_mark_written_and_rhs_enqueue (lacc);
    2830                 :            :             }
    2831                 :       6803 :           continue;
    2832                 :            :         }
    2833                 :            : 
    2834                 :     590865 :       rchild->grp_hint = 1;
    2835                 :            :       /* Because get_ref_base_and_extent always includes padding in size for
    2836                 :            :          accesses to DECLs but not necessarily for COMPONENT_REFs of the same
    2837                 :            :          type, we might be actually attempting to here to create a child of the
    2838                 :            :          same type as the parent.  */
    2839                 :     590865 :       if (!types_compatible_p (lacc->type, rchild->type))
    2840                 :     590865 :         new_acc = create_artificial_child_access (lacc, rchild, norm_offset,
    2841                 :            :                                                   false,
    2842                 :            :                                                   (lacc->grp_write
    2843                 :     590865 :                                                    || rchild->grp_write));
    2844                 :            :       else
    2845                 :            :         new_acc = lacc;
    2846                 :     590865 :       gcc_checking_assert (new_acc);
    2847                 :     590865 :       if (racc->first_child)
    2848                 :     590865 :         propagate_subaccesses_from_rhs (new_acc, rchild);
    2849                 :            : 
    2850                 :     590865 :       add_access_to_rhs_work_queue (lacc);
    2851                 :     590865 :       ret = true;
    2852                 :            :     }
    2853                 :            : 
    2854                 :            :   return ret;
    2855                 :            : }
    2856                 :            : 
    2857                 :            : /* Propagate subaccesses of LACC across an assignment link to RACC if they
    2858                 :            :    should inhibit total scalarization of the corresponding area.  No flags are
    2859                 :            :    being propagated in the process.  Return true if anything changed.  */
    2860                 :            : 
    2861                 :            : static bool
    2862                 :    4132610 : propagate_subaccesses_from_lhs (struct access *lacc, struct access *racc)
    2863                 :            : {
    2864                 :    4132610 :   if (is_gimple_reg_type (racc->type)
    2865                 :    1491810 :       || lacc->grp_unscalarizable_region
    2866                 :    5623660 :       || racc->grp_unscalarizable_region)
    2867                 :            :     return false;
    2868                 :            : 
    2869                 :            :   /* TODO: Do we want set some new racc flag to stop potential total
    2870                 :            :      scalarization if lacc is a scalar access (and none fo the two have
    2871                 :            :      children)?  */
    2872                 :            : 
    2873                 :    1491050 :   bool ret = false;
    2874                 :    1491050 :   HOST_WIDE_INT norm_delta = racc->offset - lacc->offset;
    2875                 :    1491050 :   for (struct access *lchild = lacc->first_child;
    2876                 :    4030520 :        lchild;
    2877                 :    2539470 :        lchild = lchild->next_sibling)
    2878                 :            :     {
    2879                 :    2539470 :       struct access *matching_acc = NULL;
    2880                 :    2539470 :       HOST_WIDE_INT norm_offset = lchild->offset + norm_delta;
    2881                 :            : 
    2882                 :    4703760 :       if (lchild->grp_unscalarizable_region
    2883                 :    2916320 :           || child_would_conflict_in_acc (racc, norm_offset, lchild->size,
    2884                 :    2539470 :                                           &matching_acc)
    2885                 :    2914680 :           || !budget_for_propagation_access (racc->base))
    2886                 :            :         {
    2887                 :    2164280 :           if (matching_acc
    2888                 :    2164280 :               && propagate_subaccesses_from_lhs (lchild, matching_acc))
    2889                 :        172 :             add_access_to_lhs_work_queue (matching_acc);
    2890                 :    2164280 :           continue;
    2891                 :            :         }
    2892                 :            : 
    2893                 :            :       /* Because get_ref_base_and_extent always includes padding in size for
    2894                 :            :          accesses to DECLs but not necessarily for COMPONENT_REFs of the same
    2895                 :            :          type, we might be actually attempting to here to create a child of the
    2896                 :            :          same type as the parent.  */
    2897                 :     375191 :       if (!types_compatible_p (racc->type, lchild->type))
    2898                 :            :         {
    2899                 :     375187 :           struct access *new_acc
    2900                 :     375187 :             = create_artificial_child_access (racc, lchild, norm_offset,
    2901                 :            :                                               true, false);
    2902                 :     375187 :           propagate_subaccesses_from_lhs (lchild, new_acc);
    2903                 :            :         }
    2904                 :            :       else
    2905                 :          4 :         propagate_subaccesses_from_lhs (lchild, racc);
    2906                 :            :       ret = true;
    2907                 :            :     }
    2908                 :            :   return ret;
    2909                 :            : }
    2910                 :            : 
    2911                 :            : /* Propagate all subaccesses across assignment links.  */
    2912                 :            : 
    2913                 :            : static void
    2914                 :     448277 : propagate_all_subaccesses (void)
    2915                 :            : {
    2916                 :     448277 :   propagation_budget = new hash_map<tree, unsigned>;
    2917                 :    1610380 :   while (rhs_work_queue_head)
    2918                 :            :     {
    2919                 :    1162100 :       struct access *racc = pop_access_from_rhs_work_queue ();
    2920                 :    1162100 :       struct assign_link *link;
    2921                 :            : 
    2922                 :    1162100 :       if (racc->group_representative)
    2923                 :    1162100 :         racc= racc->group_representative;
    2924                 :    1162100 :       gcc_assert (racc->first_rhs_link);
    2925                 :            : 
    2926                 :    3229800 :       for (link = racc->first_rhs_link; link; link = link->next_rhs)
    2927                 :            :         {
    2928                 :    2067700 :           struct access *lacc = link->lacc;
    2929                 :            : 
    2930                 :    2067700 :           if (!bitmap_bit_p (candidate_bitmap, DECL_UID (lacc->base)))
    2931                 :        428 :             continue;
    2932                 :    2067270 :           lacc = lacc->group_representative;
    2933                 :            : 
    2934                 :    2067270 :           bool reque_parents = false;
    2935                 :    2067270 :           if (!bitmap_bit_p (candidate_bitmap, DECL_UID (racc->base)))
    2936                 :            :             {
    2937                 :         67 :               if (!lacc->grp_write)
    2938                 :            :                 {
    2939                 :         57 :                   subtree_mark_written_and_rhs_enqueue (lacc);
    2940                 :         57 :                   reque_parents = true;
    2941                 :            :                 }
    2942                 :            :             }
    2943                 :    2067200 :           else if (propagate_subaccesses_from_rhs (lacc, racc))
    2944                 :            :             reque_parents = true;
    2945                 :            : 
    2946                 :         57 :           if (reque_parents)
    2947                 :     913913 :             do
    2948                 :            :               {
    2949                 :     913913 :                 add_access_to_rhs_work_queue (lacc);
    2950                 :     913913 :                 lacc = lacc->parent;
    2951                 :            :               }
    2952                 :     913913 :             while (lacc);
    2953                 :            :         }
    2954                 :            :     }
    2955                 :            : 
    2956                 :    1359330 :   while (lhs_work_queue_head)
    2957                 :            :     {
    2958                 :     911049 :       struct access *lacc = pop_access_from_lhs_work_queue ();
    2959                 :     911049 :       struct assign_link *link;
    2960                 :            : 
    2961                 :     911049 :       if (lacc->group_representative)
    2962                 :     911043 :         lacc = lacc->group_representative;
    2963                 :     911049 :       gcc_assert (lacc->first_lhs_link);
    2964                 :            : 
    2965                 :     911049 :       if (!bitmap_bit_p (candidate_bitmap, DECL_UID (lacc->base)))
    2966                 :        165 :         continue;
    2967                 :            : 
    2968                 :    2506430 :       for (link = lacc->first_lhs_link; link; link = link->next_lhs)
    2969                 :            :         {
    2970                 :    1595540 :           struct access *racc = link->racc;
    2971                 :            : 
    2972                 :    1595540 :           if (racc->group_representative)
    2973                 :    1595540 :             racc = racc->group_representative;
    2974                 :    1595540 :           if (!bitmap_bit_p (candidate_bitmap, DECL_UID (racc->base)))
    2975                 :         57 :             continue;
    2976                 :    1595490 :           if (propagate_subaccesses_from_lhs (lacc, racc))
    2977                 :     187716 :             add_access_to_lhs_work_queue (racc);
    2978                 :            :         }
    2979                 :            :     }
    2980                 :     896554 :   delete propagation_budget;
    2981                 :     448277 : }
    2982                 :            : 
    2983                 :            : /* Return true if the forest beginning with ROOT does not contain
    2984                 :            :    unscalarizable regions or non-byte aligned accesses.  */
    2985                 :            : 
    2986                 :            : static bool
    2987                 :     903494 : can_totally_scalarize_forest_p (struct access *root)
    2988                 :            : {
    2989                 :     903494 :   struct access *access = root;
    2990                 :    2633160 :   do
    2991                 :            :     {
    2992                 :    2633160 :       if (access->grp_unscalarizable_region
    2993                 :    2632150 :           || (access->offset % BITS_PER_UNIT) != 0
    2994                 :    2631240 :           || (access->size % BITS_PER_UNIT) != 0
    2995                 :    5263380 :           || (is_gimple_reg_type (access->type)
    2996                 :    1637550 :               && access->first_child))
    2997                 :            :         return false;
    2998                 :            : 
    2999                 :    2630100 :       if (access->first_child)
    3000                 :            :         access = access->first_child;
    3001                 :    1827200 :       else if (access->next_sibling)
    3002                 :            :         access = access->next_sibling;
    3003                 :            :       else
    3004                 :            :         {
    3005                 :    1786710 :           while (access->parent && !access->next_sibling)
    3006                 :            :             access = access->parent;
    3007                 :     985908 :           if (access->next_sibling)
    3008                 :            :             access = access->next_sibling;
    3009                 :            :           else
    3010                 :            :             {
    3011                 :     902363 :               gcc_assert (access == root);
    3012                 :     902363 :               root = root->next_grp;
    3013                 :     902363 :               access = root;
    3014                 :            :             }
    3015                 :            :         }
    3016                 :            :     }
    3017                 :    2630100 :   while (access);
    3018                 :            :   return true;
    3019                 :            : }
    3020                 :            : 
    3021                 :            : /* Create and return an ACCESS in PARENT spanning from POS with SIZE, TYPE and
    3022                 :            :    reference EXPR for total scalarization purposes and mark it as such.  Within
    3023                 :            :    the children of PARENT, link it in between PTR and NEXT_SIBLING.  */
    3024                 :            : 
    3025                 :            : static struct access *
    3026                 :     522333 : create_total_scalarization_access (struct access *parent, HOST_WIDE_INT pos,
    3027                 :            :                                    HOST_WIDE_INT size, tree type, tree expr,
    3028                 :            :                                    struct access **ptr,
    3029                 :            :                                    struct access *next_sibling)
    3030                 :            : {
    3031                 :     522333 :   struct access *access = access_pool.allocate ();
    3032                 :     522333 :   memset (access, 0, sizeof (struct access));
    3033                 :     522333 :   access->base = parent->base;
    3034                 :     522333 :   access->offset = pos;
    3035                 :     522333 :   access->size = size;
    3036                 :     522333 :   access->expr = expr;
    3037                 :     522333 :   access->type = type;
    3038                 :     522333 :   access->parent = parent;
    3039                 :     522333 :   access->grp_write = parent->grp_write;
    3040                 :     522333 :   access->grp_total_scalarization = 1;
    3041                 :     522333 :   access->grp_hint = 1;
    3042                 :     522333 :   access->grp_same_access_path = path_comparable_for_same_access (expr);
    3043                 :     522333 :   access->reverse = reverse_storage_order_for_component_p (expr);
    3044                 :            : 
    3045                 :     522333 :   access->next_sibling = next_sibling;
    3046                 :     522333 :   *ptr = access;
    3047                 :     522333 :   return access;
    3048                 :            : }
    3049                 :            : 
    3050                 :            : /* Create and return an ACCESS in PARENT spanning from POS with SIZE, TYPE and
    3051                 :            :    reference EXPR for total scalarization purposes and mark it as such, link it
    3052                 :            :    at *PTR and reshape the tree so that those elements at *PTR and their
    3053                 :            :    siblings which fall within the part described by POS and SIZE are moved to
    3054                 :            :    be children of the new access.  If a partial overlap is detected, return
    3055                 :            :    NULL.  */
    3056                 :            : 
    3057                 :            : static struct access *
    3058                 :     522333 : create_total_access_and_reshape (struct access *parent, HOST_WIDE_INT pos,
    3059                 :            :                                  HOST_WIDE_INT size, tree type, tree expr,
    3060                 :            :                                  struct access **ptr)
    3061                 :            : {
    3062                 :     522333 :   struct access **p = ptr;
    3063                 :            : 
    3064                 :     608918 :   while (*p && (*p)->offset < pos + size)
    3065                 :            :     {
    3066                 :      86585 :       if ((*p)->offset + (*p)->size > pos + size)
    3067                 :            :         return NULL;
    3068                 :      86585 :       p = &(*p)->next_sibling;
    3069                 :            :     }
    3070                 :            : 
    3071                 :     522333 :   struct access *next_child = *ptr;
    3072                 :     522333 :   struct access *new_acc
    3073                 :     522333 :     = create_total_scalarization_access (parent, pos, size, type, expr,
    3074                 :            :                                          ptr, *p);
    3075                 :     522333 :   if (p != ptr)
    3076                 :            :     {
    3077                 :      35728 :       new_acc->first_child = next_child;
    3078                 :      35728 :       *p = NULL;
    3079                 :     122313 :       for (struct access *a = next_child; a; a = a->next_sibling)
    3080                 :      86585 :         a->parent = new_acc;
    3081                 :            :     }
    3082                 :            :   return new_acc;
    3083                 :            : }
    3084                 :            : 
    3085                 :            : static bool totally_scalarize_subtree (struct access *root);
    3086                 :            : 
    3087                 :            : /* Return true if INNER is either the same type as OUTER or if it is the type
    3088                 :            :    of a record field in OUTER at offset zero, possibly in nested
    3089                 :            :    sub-records.  */
    3090                 :            : 
    3091                 :            : static bool
    3092                 :     214346 : access_and_field_type_match_p (tree outer, tree inner)
    3093                 :            : {
    3094                 :     214346 :   if (TYPE_MAIN_VARIANT (outer) == TYPE_MAIN_VARIANT (inner))
    3095                 :            :     return true;
    3096                 :         79 :   if (TREE_CODE (outer) != RECORD_TYPE)
    3097                 :            :     return false;
    3098                 :         67 :   tree fld = TYPE_FIELDS (outer);
    3099                 :        713 :   while (fld)
    3100                 :            :     {
    3101                 :        712 :      if (TREE_CODE (fld) == FIELD_DECL)
    3102                 :            :        {
    3103                 :         76 :         if (!zerop (DECL_FIELD_OFFSET (fld)))
    3104                 :            :           return false;
    3105                 :         76 :         if (TYPE_MAIN_VARIANT (TREE_TYPE (fld)) == inner)
    3106                 :            :           return true;
    3107                 :         72 :         if (TREE_CODE (TREE_TYPE (fld)) == RECORD_TYPE)
    3108                 :         10 :           fld = TYPE_FIELDS (TREE_TYPE (fld));
    3109                 :            :         else
    3110                 :            :           return false;
    3111                 :            :        }
    3112                 :            :      else
    3113                 :        636 :        fld = DECL_CHAIN (fld);
    3114                 :            :     }
    3115                 :            :   return false;
    3116                 :            : }
    3117                 :            : 
    3118                 :            : /* Return type of total_should_skip_creating_access indicating whether a total
    3119                 :            :    scalarization access for a field/element should be created, whether it
    3120                 :            :    already exists or whether the entire total scalarization has to fail.  */
    3121                 :            : 
    3122                 :            : enum total_sra_field_state {TOTAL_FLD_CREATE, TOTAL_FLD_DONE, TOTAL_FLD_FAILED};
    3123                 :            : 
    3124                 :            : /* Do all the necessary steps in total scalarization when the given aggregate
    3125                 :            :    type has a TYPE at POS with the given SIZE should be put into PARENT and
    3126                 :            :    when we have processed all its siblings with smaller offsets up until and
    3127                 :            :    including LAST_SEEN_SIBLING (which can be NULL).
    3128                 :            : 
    3129                 :            :    If some further siblings are to be skipped, set *LAST_SEEN_SIBLING as
    3130                 :            :    appropriate.  Return TOTAL_FLD_CREATE id the caller should carry on with
    3131                 :            :    creating a new access, TOTAL_FLD_DONE if access or accesses capable of
    3132                 :            :    representing the described part of the aggregate for the purposes of total
    3133                 :            :    scalarization already exist or TOTAL_FLD_FAILED if there is a problem which
    3134                 :            :    prevents total scalarization from happening at all.  */
    3135                 :            : 
    3136                 :            : static enum total_sra_field_state
    3137                 :    2230020 : total_should_skip_creating_access (struct access *parent,
    3138                 :            :                                    struct access **last_seen_sibling,
    3139                 :            :                                    tree type, HOST_WIDE_INT pos,
    3140                 :            :                                    HOST_WIDE_INT size)
    3141                 :            : {
    3142                 :    2230020 :   struct access *next_child;
    3143                 :    2230020 :   if (!*last_seen_sibling)
    3144                 :    1027520 :     next_child = parent->first_child;
    3145                 :            :   else
    3146                 :    1202500 :     next_child = (*last_seen_sibling)->next_sibling;
    3147                 :            : 
    3148                 :            :   /* First, traverse the chain of siblings until it points to an access with
    3149                 :            :      offset at least equal to POS.  Check all skipped accesses whether they
    3150                 :            :      span the POS boundary and if so, return with a failure.  */
    3151                 :    2230020 :   while (next_child && next_child->offset < pos)
    3152                 :            :     {
    3153                 :          0 :       if (next_child->offset + next_child->size > pos)
    3154                 :            :         return TOTAL_FLD_FAILED;
    3155                 :          0 :       *last_seen_sibling = next_child;
    3156                 :          0 :       next_child = next_child->next_sibling;
    3157                 :            :     }
    3158                 :            : 
    3159                 :            :   /* Now check whether next_child has exactly the right POS and SIZE and if so,
    3160                 :            :      whether it can represent what we need and can be totally scalarized
    3161                 :            :      itself.  */
    3162                 :    2230020 :   if (next_child && next_child->offset == pos
    3163                 :    1743000 :       && next_child->size == size)
    3164                 :            :     {
    3165                 :    1707570 :       if (!is_gimple_reg_type (next_child->type)
    3166                 :    1707570 :           && (!access_and_field_type_match_p (type, next_child->type)
    3167                 :     214271 :               || !totally_scalarize_subtree (next_child)))
    3168                 :         75 :         return TOTAL_FLD_FAILED;
    3169                 :            : 
    3170                 :    1707500 :       *last_seen_sibling = next_child;
    3171                 :    1707500 :       return TOTAL_FLD_DONE;
    3172                 :            :     }
    3173                 :            : 
    3174                 :            :   /* If the child we're looking at would partially overlap, we just cannot
    3175                 :            :      totally scalarize.  */
    3176                 :     522445 :   if (next_child
    3177                 :      51056 :       && next_child->offset < pos + size
    3178                 :      35840 :       && next_child->offset + next_child->size > pos + size)
    3179                 :            :     return TOTAL_FLD_FAILED;
    3180                 :            : 
    3181                 :     522375 :   if (is_gimple_reg_type (type))
    3182                 :            :     {
    3183                 :            :       /* We don't scalarize accesses that are children of other scalar type
    3184                 :            :          accesses, so if we go on and create an access for a register type,
    3185                 :            :          there should not be any pre-existing children.  There are rare cases
    3186                 :            :          where the requested type is a vector but we already have register
    3187                 :            :          accesses for all its elements which is equally good.  Detect that
    3188                 :            :          situation or whether we need to bail out.  */
    3189                 :            : 
    3190                 :            :       HOST_WIDE_INT covered = pos;
    3191                 :            :       bool skipping = false;
    3192                 :     443003 :       while (next_child
    3193                 :     443003 :              && next_child->offset + next_child->size <= pos + size)
    3194                 :            :         {
    3195                 :        168 :           if (next_child->offset != covered
    3196                 :        168 :               || !is_gimple_reg_type (next_child->type))
    3197                 :            :             return TOTAL_FLD_FAILED;
    3198                 :            : 
    3199                 :        168 :           covered += next_child->size;
    3200                 :        168 :           *last_seen_sibling = next_child;
    3201                 :        168 :           next_child = next_child->next_sibling;
    3202                 :        168 :           skipping = true;
    3203                 :            :         }
    3204                 :            : 
    3205                 :     442835 :       if (skipping)
    3206                 :            :         {
    3207                 :         42 :           if (covered != pos + size)
    3208                 :            :             return TOTAL_FLD_FAILED;
    3209                 :            :           else
    3210                 :         42 :             return TOTAL_FLD_DONE;
    3211                 :            :         }
    3212                 :            :     }
    3213                 :            : 
    3214                 :            :   return TOTAL_FLD_CREATE;
    3215                 :            : }
    3216                 :            : 
    3217                 :            : /* Go over sub-tree rooted in ROOT and attempt to create scalar accesses
    3218                 :            :    spanning all uncovered areas covered by ROOT, return false if the attempt
    3219                 :            :    failed.  All created accesses will have grp_unscalarizable_region set (and
    3220                 :            :    should be ignored if the function returns false).  */
    3221                 :            : 
    3222                 :            : static bool
    3223                 :    1055750 : totally_scalarize_subtree (struct access *root)
    3224                 :            : {
    3225                 :    1055750 :   gcc_checking_assert (!root->grp_unscalarizable_region);
    3226                 :    1055750 :   gcc_checking_assert (!is_gimple_reg_type (root->type));
    3227                 :            : 
    3228                 :    1055750 :   struct access *last_seen_sibling = NULL;
    3229                 :            : 
    3230                 :    1055750 :   switch (TREE_CODE (root->type))
    3231                 :            :     {
    3232                 :    1049320 :     case RECORD_TYPE:
    3233                 :    7067670 :       for (tree fld = TYPE_FIELDS (root->type); fld; fld = DECL_CHAIN (fld))
    3234                 :    6018500 :         if (TREE_CODE (fld) == FIELD_DECL)
    3235                 :            :           {
    3236                 :    2225900 :             tree ft = TREE_TYPE (fld);
    3237                 :    2225900 :             HOST_WIDE_INT fsize = tree_to_uhwi (DECL_SIZE (fld));
    3238                 :    2225900 :             if (!fsize)
    3239                 :       8809 :               continue;
    3240                 :            : 
    3241                 :    2217090 :             HOST_WIDE_INT pos = root->offset + int_bit_position (fld);
    3242                 :    2217090 :             enum total_sra_field_state
    3243                 :    2217090 :               state = total_should_skip_creating_access (root,
    3244                 :            :                                                          &last_seen_sibling,
    3245                 :            :                                                          ft, pos, fsize);
    3246                 :    2217090 :             switch (state)
    3247                 :            :               {
    3248                 :            :               case TOTAL_FLD_FAILED:
    3249                 :            :                 return false;
    3250                 :    1703740 :               case TOTAL_FLD_DONE:
    3251                 :    1703740 :                 continue;
    3252                 :     513209 :               case TOTAL_FLD_CREATE:
    3253                 :     513209 :                 break;
    3254                 :          0 :               default:
    3255                 :          0 :                 gcc_unreachable ();
    3256                 :            :               }
    3257                 :            : 
    3258                 :    1026420 :             struct access **p = (last_seen_sibling
    3259                 :     513209 :                                  ? &last_seen_sibling->next_sibling
    3260                 :     513209 :                                  : &root->first_child);
    3261                 :     513209 :             tree nref = build3 (COMPONENT_REF, ft, root->expr, fld, NULL_TREE);
    3262                 :     513209 :             struct access *new_child
    3263                 :     513209 :               = create_total_access_and_reshape (root, pos, fsize, ft, nref, p);
    3264                 :     513209 :             if (!new_child)
    3265                 :            :               return false;
    3266                 :            : 
    3267                 :     513209 :             if (!is_gimple_reg_type (ft)
    3268                 :     513209 :                 && !totally_scalarize_subtree (new_child))
    3269                 :            :               return false;
    3270                 :     513209 :             last_seen_sibling = new_child;
    3271                 :            :           }
    3272                 :            :       break;
    3273                 :       6431 :     case ARRAY_TYPE:
    3274                 :       6431 :       {
    3275                 :       6431 :         tree elemtype = TREE_TYPE (root->type);
    3276                 :       6431 :         tree elem_size = TYPE_SIZE (elemtype);
    3277                 :       6431 :         gcc_assert (elem_size && tree_fits_shwi_p (elem_size));
    3278                 :       6431 :         HOST_WIDE_INT el_size = tree_to_shwi (elem_size);
    3279                 :       6431 :         gcc_assert (el_size > 0);
    3280                 :            : 
    3281                 :       6431 :         tree minidx = TYPE_MIN_VALUE (TYPE_DOMAIN (root->type));
    3282                 :       6431 :         gcc_assert (TREE_CODE (minidx) == INTEGER_CST);
    3283                 :       6431 :         tree maxidx = TYPE_MAX_VALUE (TYPE_DOMAIN (root->type));
    3284                 :            :         /* Skip (some) zero-length arrays; others have MAXIDX == MINIDX - 1.  */
    3285                 :       6431 :         if (!maxidx)
    3286                 :          0 :           goto out;
    3287                 :       6431 :         gcc_assert (TREE_CODE (maxidx) == INTEGER_CST);
    3288                 :       6431 :         tree domain = TYPE_DOMAIN (root->type);
    3289                 :            :         /* MINIDX and MAXIDX are inclusive, and must be interpreted in
    3290                 :            :            DOMAIN (e.g. signed int, whereas min/max may be size_int).  */
    3291                 :       6431 :         offset_int idx = wi::to_offset (minidx);
    3292                 :       6431 :         offset_int max = wi::to_offset (maxidx);
    3293                 :       6431 :         if (!TYPE_UNSIGNED (domain))
    3294                 :            :           {
    3295                 :       6431 :             idx = wi::sext (idx, TYPE_PRECISION (domain));
    3296                 :       6431 :             max = wi::sext (max, TYPE_PRECISION (domain));
    3297                 :            :           }
    3298                 :       6431 :         for (HOST_WIDE_INT pos = root->offset;
    3299                 :      19355 :              idx <= max;
    3300                 :      12924 :              pos += el_size, ++idx)
    3301                 :            :           {
    3302                 :      12924 :             enum total_sra_field_state
    3303                 :      12924 :               state = total_should_skip_creating_access (root,
    3304                 :            :                                                          &last_seen_sibling,
    3305                 :            :                                                          elemtype, pos,
    3306                 :            :                                                          el_size);
    3307                 :      12924 :             switch (state)
    3308                 :            :               {
    3309                 :            :               case TOTAL_FLD_FAILED:
    3310                 :          0 :                 return false;
    3311                 :       3800 :               case TOTAL_FLD_DONE:
    3312                 :       3800 :                 continue;
    3313                 :       9124 :               case TOTAL_FLD_CREATE:
    3314                 :       9124 :                 break;
    3315                 :          0 :               default:
    3316                 :          0 :                 gcc_unreachable ();
    3317                 :            :               }
    3318                 :            : 
    3319                 :      18248 :             struct access **p = (last_seen_sibling
    3320                 :       9124 :                                  ? &last_seen_sibling->next_sibling
    3321                 :       9124 :                                  : &root->first_child);
    3322                 :       9124 :             tree nref = build4 (ARRAY_REF, elemtype, root->expr,
    3323                 :            :                                 wide_int_to_tree (domain, idx),
    3324                 :            :                                 NULL_TREE, NULL_TREE);
    3325                 :       9124 :             struct access *new_child
    3326                 :       9124 :               = create_total_access_and_reshape (root, pos, el_size, elemtype,
    3327                 :            :                                                  nref, p);
    3328                 :       9124 :             if (!new_child)
    3329                 :            :               return false;
    3330                 :            : 
    3331                 :       9124 :             if (!is_gimple_reg_type (elemtype)
    3332                 :       9124 :                 && !totally_scalarize_subtree (new_child))
    3333                 :            :               return false;
    3334                 :       9124 :             last_seen_sibling = new_child;
    3335                 :            :           }
    3336                 :            :       }
    3337                 :       6431 :       break;
    3338                 :          0 :     default:
    3339                 :          0 :       gcc_unreachable ();
    3340                 :            :     }
    3341                 :            : 
    3342                 :            :  out:
    3343                 :            :   return true;
    3344                 :            : }
    3345                 :            : 
    3346                 :            : /* Go through all accesses collected throughout the (intraprocedural) analysis
    3347                 :            :    stage, exclude overlapping ones, identify representatives and build trees
    3348                 :            :    out of them, making decisions about scalarization on the way.  Return true
    3349                 :            :    iff there are any to-be-scalarized variables after this stage. */
    3350                 :            : 
    3351                 :            : static bool
    3352                 :     448277 : analyze_all_variable_accesses (void)
    3353                 :            : {
    3354                 :     448277 :   int res = 0;
    3355                 :     448277 :   bitmap tmp = BITMAP_ALLOC (NULL);
    3356                 :     448277 :   bitmap_iterator bi;
    3357                 :     448277 :   unsigned i;
    3358                 :            : 
    3359                 :     448277 :   bitmap_copy (tmp, candidate_bitmap);
    3360                 :    3431340 :   EXECUTE_IF_SET_IN_BITMAP (tmp, 0, i, bi)
    3361                 :            :     {
    3362                 :    2983060 :       tree var = candidate (i);
    3363                 :    2983060 :       struct access *access;
    3364                 :            : 
    3365                 :    2983060 :       access = sort_and_splice_var_accesses (var);
    3366                 :    5840300 :       if (!access || !build_access_trees (access))
    3367                 :     126603 :         disqualify_candidate (var,
    3368                 :            :                               "No or inhibitingly overlapping accesses.");
    3369                 :            :     }
    3370                 :            : 
    3371                 :     448277 :   propagate_all_subaccesses ();
    3372                 :            : 
    3373                 :     448277 :   bool optimize_speed_p = !optimize_function_for_size_p (cfun);
    3374                 :            :   /* If the user didn't set PARAM_SRA_MAX_SCALARIZATION_SIZE_<...>,
    3375                 :            :      fall back to a target default.  */
    3376                 :     448277 :   unsigned HOST_WIDE_INT max_scalarization_size
    3377                 :     448277 :     = get_move_ratio (optimize_speed_p) * UNITS_PER_WORD;
    3378                 :            : 
    3379                 :     448277 :   if (optimize_speed_p)
    3380                 :            :     {
    3381                 :     438015 :       if (global_options_set.x_param_sra_max_scalarization_size_speed)
    3382                 :          5 :         max_scalarization_size = param_sra_max_scalarization_size_speed;
    3383                 :            :     }
    3384                 :            :   else
    3385                 :            :     {
    3386                 :      10262 :       if (global_options_set.x_param_sra_max_scalarization_size_size)
    3387                 :          0 :         max_scalarization_size = param_sra_max_scalarization_size_size;
    3388                 :            :     }
    3389                 :     448277 :   max_scalarization_size *= BITS_PER_UNIT;
    3390                 :            : 
    3391                 :    3304730 :   EXECUTE_IF_SET_IN_BITMAP (candidate_bitmap, 0, i, bi)
    3392                 :    2856460 :     if (bitmap_bit_p (should_scalarize_away_bitmap, i)
    3393                 :    2856460 :         && !bitmap_bit_p (cannot_scalarize_away_bitmap, i))
    3394                 :            :       {
    3395                 :     971007 :         tree var = candidate (i);
    3396                 :     971007 :         if (!VAR_P (var))
    3397                 :      56437 :           continue;
    3398                 :            : 
    3399                 :     914570 :         if (tree_to_uhwi (TYPE_SIZE (TREE_TYPE (var))) > max_scalarization_size)
    3400                 :            :           {
    3401                 :      11929 :             if (dump_file && (dump_flags & TDF_DETAILS))
    3402                 :            :               {
    3403                 :          0 :                 fprintf (dump_file, "Too big to totally scalarize: ");
    3404                 :          0 :                 print_generic_expr (dump_file, var);
    3405                 :          0 :                 fprintf (dump_file, " (UID: %u)\n", DECL_UID (var));
    3406                 :            :               }
    3407                 :      11929 :             continue;
    3408                 :            :           }
    3409                 :            : 
    3410                 :     902641 :         bool all_types_ok = true;
    3411                 :     902641 :         for (struct access *access = get_first_repr_for_decl (var);
    3412                 :    1793490 :              access;
    3413                 :     890847 :              access = access->next_grp)
    3414                 :     903494 :           if (!can_totally_scalarize_forest_p (access)
    3415                 :    1803930 :               || !scalarizable_type_p (access->type, constant_decl_p (var)))
    3416                 :            :             {
    3417                 :            :               all_types_ok = false;
    3418                 :            :               break;
    3419                 :            :             }
    3420                 :     902641 :         if (!all_types_ok)
    3421                 :      12647 :           continue;
    3422                 :            : 
    3423                 :     889994 :         if (dump_file && (dump_flags & TDF_DETAILS))
    3424                 :            :           {
    3425                 :          4 :             fprintf (dump_file, "Will attempt to totally scalarize ");
    3426                 :          4 :             print_generic_expr (dump_file, var);
    3427                 :          4 :             fprintf (dump_file, " (UID: %u): \n", DECL_UID (var));
    3428                 :            :           }
    3429                 :     889994 :         bool scalarized = true;
    3430                 :     889994 :         for (struct access *access = get_first_repr_for_decl (var);
    3431                 :    1780700 :              access;
    3432                 :     890701 :              access = access->next_grp)
    3433                 :     890846 :           if (!is_gimple_reg_type (access->type)
    3434                 :     890846 :               && !totally_scalarize_subtree (access))
    3435                 :            :             {
    3436                 :            :               scalarized = false;
    3437                 :            :               break;
    3438                 :            :             }
    3439                 :            : 
    3440                 :     889994 :         if (scalarized)
    3441                 :     889849 :           for (struct access *access = get_first_repr_for_decl (var);
    3442                 :    1780550 :                access;
    3443                 :     890701 :                access = access->next_grp)
    3444                 :     890701 :             access->grp_total_scalarization = true;
    3445                 :            :       }
    3446                 :            : 
    3447                 :     448277 :   if (flag_checking)
    3448                 :     448277 :     verify_all_sra_access_forests ();
    3449                 :            : 
    3450                 :     448277 :   bitmap_copy (tmp, candidate_bitmap);
    3451                 :    3304730 :   EXECUTE_IF_SET_IN_BITMAP (tmp, 0, i, bi)
    3452                 :            :     {
    3453                 :    2856460 :       tree var = candidate (i);
    3454                 :    2856460 :       struct access *access = get_first_repr_for_decl (var);
    3455                 :            : 
    3456                 :    2856460 :       if (analyze_access_trees (access))
    3457                 :            :         {
    3458                 :    1398340 :           res++;
    3459                 :    1398340 :           if (dump_file && (dump_flags & TDF_DETAILS))
    3460                 :            :             {
    3461                 :          9 :               fprintf (dump_file, "\nAccess trees for ");
    3462                 :          9 :               print_generic_expr (dump_file, var);
    3463                 :          9 :               fprintf (dump_file, " (UID: %u): \n", DECL_UID (var));
    3464                 :          9 :               dump_access_tree (dump_file, access);
    3465                 :          9 :               fprintf (dump_file, "\n");
    3466                 :            :             }
    3467                 :            :         }
    3468                 :            :       else
    3469                 :    1458110 :         disqualify_candidate (var, "No scalar replacements to be created.");
    3470                 :            :     }
    3471                 :            : 
    3472                 :     448277 :   BITMAP_FREE (tmp);
    3473                 :            : 
    3474                 :     448277 :   if (res)
    3475                 :            :     {
    3476                 :     261601 :       statistics_counter_event (cfun, "Scalarized aggregates", res);
    3477                 :     261601 :       return true;
    3478                 :            :     }
    3479                 :            :   else
    3480                 :            :     return false;
    3481                 :            : }
    3482                 :            : 
    3483                 :            : /* Generate statements copying scalar replacements of accesses within a subtree
    3484                 :            :    into or out of AGG.  ACCESS, all its children, siblings and their children
    3485                 :            :    are to be processed.  AGG is an aggregate type expression (can be a
    3486                 :            :    declaration but does not have to be, it can for example also be a mem_ref or
    3487                 :            :    a series of handled components).  TOP_OFFSET is the offset of the processed
    3488                 :            :    subtree which has to be subtracted from offsets of individual accesses to
    3489                 :            :    get corresponding offsets for AGG.  If CHUNK_SIZE is non-null, copy only
    3490                 :            :    replacements in the interval <start_offset, start_offset + chunk_size>,
    3491                 :            :    otherwise copy all.  GSI is a statement iterator used to place the new
    3492                 :            :    statements.  WRITE should be true when the statements should write from AGG
    3493                 :            :    to the replacement and false if vice versa.  if INSERT_AFTER is true, new
    3494                 :            :    statements will be added after the current statement in GSI, they will be
    3495                 :            :    added before the statement otherwise.  */
    3496                 :            : 
    3497                 :            : static void
    3498                 :    1264290 : generate_subtree_copies (struct access *access, tree agg,
    3499                 :            :                          HOST_WIDE_INT top_offset,
    3500                 :            :                          HOST_WIDE_INT start_offset, HOST_WIDE_INT chunk_size,
    3501                 :            :                          gimple_stmt_iterator *gsi, bool write,
    3502                 :            :                          bool insert_after, location_t loc)
    3503                 :            : {
    3504                 :            :   /* Never write anything into constant pool decls.  See PR70602.  */
    3505                 :    1264290 :   if (!write && constant_decl_p (agg))
    3506                 :            :     return;
    3507                 :    2867530 :   do
    3508                 :            :     {
    3509                 :    2867530 :       if (chunk_size && access->offset >= start_offset + chunk_size)
    3510                 :            :         return;
    3511                 :            : 
    3512                 :    2865890 :       if (access->grp_to_be_replaced
    3513                 :    2497980 :           && (chunk_size == 0
    3514                 :    2867650 :               || access->offset + access->size > start_offset))
    3515                 :            :         {
    3516                 :    2497340 :           tree expr, repl = get_access_replacement (access);
    3517                 :    2497340 :           gassign *stmt;
    3518                 :            : 
    3519                 :    2497340 :           expr = build_ref_for_model (loc, agg, access->offset - top_offset,
    3520                 :            :                                       access, gsi, insert_after);
    3521                 :            : 
    3522                 :    2497340 :           if (write)
    3523                 :            :             {
    3524                 :    1129160 :               if (access->grp_partial_lhs)
    3525                 :          0 :                 expr = force_gimple_operand_gsi (gsi, expr, true, NULL_TREE,
    3526                 :          0 :                                                  !insert_after,
    3527                 :            :                                                  insert_after ? GSI_NEW_STMT
    3528                 :            :                                                  : GSI_SAME_STMT);
    3529                 :    1129160 :               stmt = gimple_build_assign (repl, expr);
    3530                 :            :             }
    3531                 :            :           else
    3532                 :            :             {
    3533                 :    1368180 :               TREE_NO_WARNING (repl) = 1;
    3534                 :    1368180 :               if (access->grp_partial_lhs)
    3535                 :         55 :                 repl = force_gimple_operand_gsi (gsi, repl, true, NULL_TREE,
    3536                 :         55 :                                                  !insert_after,
    3537                 :            :                                                  insert_after ? GSI_NEW_STMT
    3538                 :            :                                                  : GSI_SAME_STMT);
    3539                 :    1368180 :               stmt = gimple_build_assign (expr, repl);
    3540                 :            :             }
    3541                 :    2497340 :           gimple_set_location (stmt, loc);
    3542                 :            : 
    3543                 :    2497340 :           if (insert_after)
    3544                 :    1129160 :             gsi_insert_after (gsi, stmt, GSI_NEW_STMT);
    3545                 :            :           else
    3546                 :    1368180 :             gsi_insert_before (gsi, stmt, GSI_SAME_STMT);
    3547                 :    2497340 :           update_stmt (stmt);
    3548                 :    2497340 :           sra_stats.subtree_copies++;
    3549                 :            :         }
    3550                 :     368548 :       else if (write
    3551                 :     145127 :                && access->grp_to_be_debug_replaced
    3552                 :        235 :                && (chunk_size == 0
    3553                 :          0 :                    || access->offset + access->size > start_offset))
    3554                 :            :         {
    3555                 :        235 :           gdebug *ds;
    3556                 :        470 :           tree drhs = build_debug_ref_for_model (loc, agg,
    3557                 :        235 :                                                  access->offset - top_offset,
    3558                 :            :                                                  access);
    3559                 :        235 :           ds = gimple_build_debug_bind (get_access_replacement (access),
    3560                 :            :                                         drhs, gsi_stmt (*gsi));
    3561                 :        235 :           if (insert_after)
    3562                 :        235 :             gsi_insert_after (gsi, ds, GSI_NEW_STMT);
    3563                 :            :           else
    3564                 :          0 :             gsi_insert_before (gsi, ds, GSI_SAME_STMT);
    3565                 :            :         }
    3566                 :            : 
    3567                 :    2865890 :       if (access->first_child)
    3568                 :     282525 :         generate_subtree_copies (access->first_child, agg, top_offset,
    3569                 :            :                                  start_offset, chunk_size, gsi,
    3570                 :            :                                  write, insert_after, loc);
    3571                 :            : 
    3572                 :    2865890 :       access = access->next_sibling;
    3573                 :            :     }
    3574                 :    2865890 :   while (access);
    3575                 :            : }
    3576                 :            : 
    3577                 :            : /* Assign zero to all scalar replacements in an access subtree.  ACCESS is the
    3578                 :            :    root of the subtree to be processed.  GSI is the statement iterator used
    3579                 :            :    for inserting statements which are added after the current statement if
    3580                 :            :    INSERT_AFTER is true or before it otherwise.  */
    3581                 :            : 
    3582                 :            : static void
    3583                 :     421328 : init_subtree_with_zero (struct access *access, gimple_stmt_iterator *gsi,
    3584                 :            :                         bool insert_after, location_t loc)
    3585                 :            : 
    3586                 :            : {
    3587                 :     421328 :   struct access *child;
    3588                 :            : 
    3589                 :     421328 :   if (access->grp_to_be_replaced)
    3590                 :            :     {
    3591                 :     236947 :       gassign *stmt;
    3592                 :            : 
    3593                 :     236947 :       stmt = gimple_build_assign (get_access_replacement (access),
    3594                 :            :                                   build_zero_cst (access->type));
    3595                 :     236947 :       if (insert_after)
    3596                 :      25751 :         gsi_insert_after (gsi, stmt, GSI_NEW_STMT);
    3597                 :            :       else
    3598                 :     211196 :         gsi_insert_before (gsi, stmt, GSI_SAME_STMT);
    3599                 :     236947 :       update_stmt (stmt);
    3600                 :     236947 :       gimple_set_location (stmt, loc);
    3601                 :            :     }
    3602                 :     184381 :   else if (access->grp_to_be_debug_replaced)
    3603                 :            :     {
    3604                 :      11156 :       gdebug *ds
    3605                 :      11156 :         = gimple_build_debug_bind (get_access_replacement (access),
    3606                 :            :                                    build_zero_cst (access->type),
    3607                 :            :                                    gsi_stmt (*gsi));
    3608                 :      11156 :       if (insert_after)
    3609                 :      11156 :         gsi_insert_after (gsi, ds, GSI_NEW_STMT);
    3610                 :            :       else
    3611                 :          0 :         gsi_insert_before (gsi, ds, GSI_SAME_STMT);
    3612                 :            :     }
    3613                 :            : 
    3614                 :     740488 :   for (child = access->first_child; child; child = child->next_sibling)
    3615                 :     319160 :     init_subtree_with_zero (child, gsi, insert_after, loc);
    3616                 :     421328 : }
    3617                 :            : 
    3618                 :            : /* Clobber all scalar replacements in an access subtree.  ACCESS is the
    3619                 :            :    root of the subtree to be processed.  GSI is the statement iterator used
    3620                 :            :    for inserting statements which are added after the current statement if
    3621                 :            :    INSERT_AFTER is true or before it otherwise.  */
    3622                 :            : 
    3623                 :            : static void
    3624                 :    1689730 : clobber_subtree (struct access *access, gimple_stmt_iterator *gsi,
    3625                 :            :                 bool insert_after, location_t loc)
    3626                 :            : 
    3627                 :            : {
    3628                 :    1689730 :   struct access *child;
    3629                 :            : 
    3630                 :    1689730 :   if (access->grp_to_be_replaced)
    3631                 :            :     {
    3632                 :    1155140 :       tree rep = get_access_replacement (access);
    3633                 :    1155140 :       tree clobber = build_clobber (access->type);
    3634                 :    1155140 :       gimple *stmt = gimple_build_assign (rep, clobber);
    3635                 :            : 
    3636                 :    1155140 :       if (insert_after)
    3637                 :     139689 :         gsi_insert_after (gsi, stmt, GSI_NEW_STMT);
    3638                 :            :       else
    3639                 :    1015450 :         gsi_insert_before (gsi, stmt, GSI_SAME_STMT);
    3640                 :    1155140 :       update_stmt (stmt);
    3641                 :    1155140 :       gimple_set_location (stmt, loc);
    3642                 :            :     }
    3643                 :            : 
    3644                 :    2756550 :   for (child = access->first_child; child; child = child->next_sibling)
    3645                 :    1066820 :     clobber_subtree (child, gsi, insert_after, loc);
    3646                 :    1689730 : }
    3647                 :            : 
    3648                 :            : /* Search for an access representative for the given expression EXPR and
    3649                 :            :    return it or NULL if it cannot be found.  */
    3650                 :            : 
    3651                 :            : static struct access *
    3652                 :   29911600 : get_access_for_expr (tree expr)
    3653                 :            : {
    3654                 :   29911600 :   poly_int64 poffset, psize, pmax_size;
    3655                 :   29911600 :   HOST_WIDE_INT offset, max_size;
    3656                 :   29911600 :   tree base;
    3657                 :   29911600 :   bool reverse;
    3658                 :            : 
    3659                 :            :   /* FIXME: This should not be necessary but Ada produces V_C_Es with a type of
    3660                 :            :      a different size than the size of its argument and we need the latter
    3661                 :            :      one.  */
    3662                 :   29911600 :   if (TREE_CODE (expr) == VIEW_CONVERT_EXPR)
    3663                 :     209278 :     expr = TREE_OPERAND (expr, 0);
    3664                 :            : 
    3665                 :   29911600 :   base = get_ref_base_and_extent (expr, &poffset, &psize, &pmax_size,
    3666                 :            :                                   &reverse);
    3667                 :   59744800 :   if (!known_size_p (pmax_size)
    3668                 :   29833300 :       || !pmax_size.is_constant (&max_size)
    3669                 :   29833300 :       || !poffset.is_constant (&offset)
    3670                 :   29911600 :       || !DECL_P (base))
    3671                 :            :     return NULL;
    3672                 :            : 
    3673                 :   12876900 :   if (tree basesize = DECL_SIZE (base))
    3674                 :            :     {
    3675                 :   12874000 :       poly_int64 sz;
    3676                 :   12874000 :       if (offset < 0
    3677                 :   12874000 :           || !poly_int_tree_p (basesize, &sz)
    3678                 :   25748100 :           || known_le (sz, offset))
    3679                 :        794 :         return NULL;
    3680                 :            :     }
    3681                 :            : 
    3682                 :   12876100 :   if (max_size == 0
    3683                 :   25752100 :       || !bitmap_bit_p (candidate_bitmap, DECL_UID (base)))
    3684                 :    6638380 :     return NULL;
    3685                 :            : 
    3686                 :    6237690 :   return get_var_base_offset_size_access (base, offset, max_size);
    3687                 :            : }
    3688                 :            : 
    3689                 :            : /* Replace the expression EXPR with a scalar replacement if there is one and
    3690                 :            :    generate other statements to do type conversion or subtree copying if
    3691                 :            :    necessary.  GSI is used to place newly created statements, WRITE is true if
    3692                 :            :    the expression is being written to (it is on a LHS of a statement or output
    3693                 :            :    in an assembly statement).  */
    3694                 :            : 
    3695                 :            : static bool
    3696                 :    6721920 : sra_modify_expr (tree *expr, gimple_stmt_iterator *gsi, bool write)
    3697                 :            : {
    3698                 :    6721920 :   location_t loc;
    3699                 :    6721920 :   struct access *access;
    3700                 :    6721920 :   tree type, bfr, orig_expr;
    3701                 :            : 
    3702                 :    6721920 :   if (TREE_CODE (*expr) == BIT_FIELD_REF)
    3703                 :            :     {
    3704                 :       5076 :       bfr = *expr;
    3705                 :       5076 :       expr = &TREE_OPERAND (*expr, 0);
    3706                 :            :     }
    3707                 :            :   else
    3708                 :            :     bfr = NULL_TREE;
    3709                 :            : 
    3710                 :    6721920 :   if (TREE_CODE (*expr) == REALPART_EXPR || TREE_CODE (*expr) == IMAGPART_EXPR)
    3711                 :       6052 :     expr = &TREE_OPERAND (*expr, 0);
    3712                 :    6721920 :   access = get_access_for_expr (*expr);
    3713                 :    6721920 :   if (!access)
    3714                 :            :     return false;
    3715                 :     117191 :   type = TREE_TYPE (*expr);
    3716                 :     117191 :   orig_expr = *expr;
    3717                 :            : 
    3718                 :     117191 :   loc = gimple_location (gsi_stmt (*gsi));
    3719                 :     117191 :   gimple_stmt_iterator alt_gsi = gsi_none ();
    3720                 :     117191 :   if (write && stmt_ends_bb_p (gsi_stmt (*gsi)))
    3721                 :            :     {
    3722                 :      25554 :       alt_gsi = gsi_start_edge (single_non_eh_succ (gsi_bb (*gsi)));
    3723                 :      25554 :       gsi = &alt_gsi;
    3724                 :            :     }
    3725                 :            : 
    3726                 :     117191 :   if (access->grp_to_be_replaced)
    3727                 :            :     {
    3728                 :      13428 :       tree repl = get_access_replacement (access);
    3729                 :            :       /* If we replace a non-register typed access simply use the original
    3730                 :            :          access expression to extract the scalar component afterwards.
    3731                 :            :          This happens if scalarizing a function return value or parameter
    3732                 :            :          like in gcc.c-torture/execute/20041124-1.c, 20050316-1.c and
    3733                 :            :          gcc.c-torture/compile/20011217-1.c.
    3734                 :            : 
    3735                 :            :          We also want to use this when accessing a complex or vector which can
    3736                 :            :          be accessed as a different type too, potentially creating a need for
    3737                 :            :          type conversion (see PR42196) and when scalarized unions are involved
    3738                 :            :          in assembler statements (see PR42398).  */
    3739                 :      13428 :       if (!useless_type_conversion_p (type, access->type))
    3740                 :            :         {
    3741                 :      11321 :           tree ref;
    3742                 :            : 
    3743                 :      11321 :           ref = build_ref_for_model (loc, orig_expr, 0, access, gsi, false);
    3744                 :            : 
    3745                 :      11321 :           if (write)
    3746                 :            :             {
    3747                 :       3449 :               gassign *stmt;
    3748                 :            : 
    3749                 :       3449 :               if (access->grp_partial_lhs)
    3750                 :          6 :                 ref = force_gimple_operand_gsi (gsi, ref, true, NULL_TREE,
    3751                 :            :                                                  false, GSI_NEW_STMT);
    3752                 :       3449 :               stmt = gimple_build_assign (repl, ref);
    3753                 :       3449 :               gimple_set_location (stmt, loc);
    3754                 :       3449 :               gsi_insert_after (gsi, stmt, GSI_NEW_STMT);
    3755                 :            :             }
    3756                 :            :           else
    3757                 :            :             {
    3758                 :       7872 :               gassign *stmt;
    3759                 :            : 
    3760                 :       7872 :               if (access->grp_partial_lhs)
    3761                 :          6 :                 repl = force_gimple_operand_gsi (gsi, repl, true, NULL_TREE,
    3762                 :            :                                                  true, GSI_SAME_STMT);
    3763                 :       7872 :               stmt = gimple_build_assign (ref, repl);
    3764                 :       7872 :               gimple_set_location (stmt, loc);
    3765                 :       7872 :               gsi_insert_before (gsi, stmt, GSI_SAME_STMT);
    3766                 :            :             }
    3767                 :            :         }
    3768                 :            :       else
    3769                 :       2107 :         *expr = repl;
    3770                 :      13428 :       sra_stats.exprs++;
    3771                 :            :     }
    3772                 :     103763 :   else if (write && access->grp_to_be_debug_replaced)
    3773                 :            :     {
    3774                 :          6 :       gdebug *ds = gimple_build_debug_bind (get_access_replacement (access),
    3775                 :            :                                             NULL_TREE,
    3776                 :            :                                             gsi_stmt (*gsi));
    3777                 :          6 :       gsi_insert_after (gsi, ds, GSI_NEW_STMT);
    3778                 :            :     }
    3779                 :            : 
    3780                 :     117191 :   if (access->first_child)
    3781                 :            :     {
    3782                 :     101425 :       HOST_WIDE_INT start_offset, chunk_size;
    3783                 :     101425 :       if (bfr
    3784                 :       2348 :           && tree_fits_uhwi_p (TREE_OPERAND (bfr, 1))
    3785                 :     103773 :           && tree_fits_uhwi_p (TREE_OPERAND (bfr, 2)))
    3786                 :            :         {
    3787                 :       2348 :           chunk_size = tree_to_uhwi (TREE_OPERAND (bfr, 1));
    3788                 :       4696 :           start_offset = access->offset
    3789                 :       2348 :             + tree_to_uhwi (TREE_OPERAND (bfr, 2));
    3790                 :            :         }
    3791                 :            :       else
    3792                 :            :         start_offset = chunk_size = 0;
    3793                 :            : 
    3794                 :     101425 :       generate_subtree_copies (access->first_child, orig_expr, access->offset,
    3795                 :            :                                start_offset, chunk_size, gsi, write, write,
    3796                 :            :                                loc);
    3797                 :            :     }
    3798                 :            :   return true;
    3799                 :            : }
    3800                 :            : 
    3801                 :            : /* Where scalar replacements of the RHS have been written to when a replacement
    3802                 :            :    of a LHS of an assigments cannot be direclty loaded from a replacement of
    3803                 :            :    the RHS. */
    3804                 :            : enum unscalarized_data_handling { SRA_UDH_NONE,  /* Nothing done so far. */
    3805                 :            :                                   SRA_UDH_RIGHT, /* Data flushed to the RHS. */
    3806                 :            :                                   SRA_UDH_LEFT }; /* Data flushed to the LHS. */
    3807                 :            : 
    3808                 :            : struct subreplacement_assignment_data
    3809                 :            : {
    3810                 :            :   /* Offset of the access representing the lhs of the assignment.  */
    3811                 :            :   HOST_WIDE_INT left_offset;
    3812                 :            : 
    3813                 :            :   /* LHS and RHS of the original assignment.  */
    3814                 :            :   tree assignment_lhs, assignment_rhs;
    3815                 :            : 
    3816                 :            :   /* Access representing the rhs of the whole assignment.  */
    3817                 :            :   struct access *top_racc;
    3818                 :            : 
    3819                 :            :   /* Stmt iterator used for statement insertions after the original assignment.
    3820                 :            :    It points to the main GSI used to traverse a BB during function body
    3821                 :            :    modification.  */
    3822                 :            :   gimple_stmt_iterator *new_gsi;
    3823                 :            : 
    3824                 :            :   /* Stmt iterator used for statement insertions before the original
    3825                 :            :    assignment.  Keeps on pointing to the original statement.  */
    3826                 :            :   gimple_stmt_iterator old_gsi;
    3827                 :            : 
    3828                 :            :   /* Location of the assignment.   */
    3829                 :            :   location_t loc;
    3830                 :            : 
    3831                 :            :   /* Keeps the information whether we have needed to refresh replacements of
    3832                 :            :    the LHS and from which side of the assignments this takes place.  */
    3833                 :            :   enum unscalarized_data_handling refreshed;
    3834                 :            : };
    3835                 :            : 
    3836                 :            : /* Store all replacements in the access tree rooted in TOP_RACC either to their
    3837                 :            :    base aggregate if there are unscalarized data or directly to LHS of the
    3838                 :            :    statement that is pointed to by GSI otherwise.  */
    3839                 :            : 
    3840                 :            : static void
    3841                 :      41573 : handle_unscalarized_data_in_subtree (struct subreplacement_assignment_data *sad)
    3842                 :            : {
    3843                 :      41573 :   tree src;
    3844                 :      41573 :   if (sad->top_racc->grp_unscalarized_data)
    3845                 :            :     {
    3846                 :      12412 :       src = sad->assignment_rhs;
    3847                 :      12412 :       sad->refreshed = SRA_UDH_RIGHT;
    3848                 :            :     }
    3849                 :            :   else
    3850                 :            :     {
    3851                 :      29161 :       src = sad->assignment_lhs;
    3852                 :      29161 :       sad->refreshed = SRA_UDH_LEFT;
    3853                 :            :     }
    3854                 :      41573 :   generate_subtree_copies (sad->top_racc->first_child, src,
    3855                 :            :                            sad->top_racc->offset, 0, 0,
    3856                 :            :                            &sad->old_gsi, false, false, sad->loc);
    3857                 :      41573 : }
    3858                 :            : 
    3859                 :            : /* Try to generate statements to load all sub-replacements in an access subtree
    3860                 :            :    formed by children of LACC from scalar replacements in the SAD->top_racc
    3861                 :            :    subtree.  If that is not possible, refresh the SAD->top_racc base aggregate
    3862                 :            :    and load the accesses from it.  */
    3863                 :            : 
    3864                 :            : static void
    3865                 :     545496 : load_assign_lhs_subreplacements (struct access *lacc,
    3866                 :            :                                  struct subreplacement_assignment_data *sad)
    3867                 :            : {
    3868                 :    1770800 :   for (lacc = lacc->first_child; lacc; lacc = lacc->next_sibling)
    3869                 :            :     {
    3870                 :    1225310 :       HOST_WIDE_INT offset;
    3871                 :    1225310 :       offset = lacc->offset - sad->left_offset + sad->top_racc->offset;
    3872                 :            : 
    3873                 :    1225310 :       if (lacc->grp_to_be_replaced)
    3874                 :            :         {
    3875                 :    1151350 :           struct access *racc;
    3876                 :    1151350 :           gassign *stmt;
    3877                 :    1151350 :           tree rhs;
    3878                 :            : 
    3879                 :    1151350 :           racc = find_access_in_subtree (sad->top_racc, offset, lacc->size);
    3880                 :    1151350 :           if (racc && racc->grp_to_be_replaced)
    3881                 :            :             {
    3882                 :    1140850 :               rhs = get_access_replacement (racc);
    3883                 :    1140850 :               if (!useless_type_conversion_p (lacc->type, racc->type))
    3884                 :         32 :                 rhs = fold_build1_loc (sad->loc, VIEW_CONVERT_EXPR,
    3885                 :            :                                        lacc->type, rhs);
    3886                 :            : 
    3887                 :    1140850 :               if (racc->grp_partial_lhs && lacc->grp_partial_lhs)
    3888                 :          0 :                 rhs = force_gimple_operand_gsi (&sad->old_gsi, rhs, true,
    3889                 :            :                                                 NULL_TREE, true, GSI_SAME_STMT);
    3890                 :            :             }
    3891                 :            :           else
    3892                 :            :             {
    3893                 :            :               /* No suitable access on the right hand side, need to load from
    3894                 :            :                  the aggregate.  See if we have to update it first... */
    3895                 :      10493 :               if (sad->refreshed == SRA_UDH_NONE)
    3896                 :       6844 :                 handle_unscalarized_data_in_subtree (sad);
    3897                 :            : 
    3898                 :      10493 :               if (sad->refreshed == SRA_UDH_LEFT)
    3899                 :        118 :                 rhs = build_ref_for_model (sad->loc, sad->assignment_lhs,
    3900                 :        118 :                                            lacc->offset - sad->left_offset,
    3901                 :            :                                            lacc, sad->new_gsi, true);
    3902                 :            :               else
    3903                 :      10375 :                 rhs = build_ref_for_model (sad->loc, sad->assignment_rhs,
    3904                 :      10375 :                                            lacc->offset - sad->left_offset,
    3905                 :            :                                            lacc, sad->new_gsi, true);
    3906                 :      10493 :               if (lacc->grp_partial_lhs)
    3907                 :          1 :                 rhs = force_gimple_operand_gsi (sad->new_gsi,
    3908                 :            :                                                 rhs, true, NULL_TREE,
    3909                 :            :                                                 false, GSI_NEW_STMT);
    3910                 :            :             }
    3911                 :            : 
    3912                 :    1151350 :           stmt = gimple_build_assign (get_access_replacement (lacc), rhs);
    3913                 :    1151350 :           gsi_insert_after (sad->new_gsi, stmt, GSI_NEW_STMT);
    3914                 :    1151350 :           gimple_set_location (stmt, sad->loc);
    3915                 :    1151350 :           update_stmt (stmt);
    3916                 :    1151350 :           sra_stats.subreplacements++;
    3917                 :            :         }
    3918                 :            :       else
    3919                 :            :         {
    3920                 :      73962 :           if (sad->refreshed == SRA_UDH_NONE
    3921                 :      32710 :               && lacc->grp_read && !lacc->grp_covered)
    3922                 :          0 :             handle_unscalarized_data_in_subtree (sad);
    3923                 :            : 
    3924                 :      73962 :           if (lacc && lacc->grp_to_be_debug_replaced)
    3925                 :            :             {
    3926                 :      25639 :               gdebug *ds;
    3927                 :      25639 :               tree drhs;
    3928                 :      25639 :               struct access *racc = find_access_in_subtree (sad->top_racc,
    3929                 :            :                                                             offset,
    3930                 :            :                                                             lacc->size);
    3931                 :            : 
    3932                 :      25639 :               if (racc && racc->grp_to_be_replaced)
    3933                 :            :                 {
    3934                 :      25637 :                   if (racc->grp_write || constant_decl_p (racc->base))
    3935                 :      24690 :                     drhs = get_access_replacement (racc);
    3936                 :            :                   else
    3937                 :            :                     drhs = NULL;
    3938                 :            :                 }
    3939                 :          2 :               else if (sad->refreshed == SRA_UDH_LEFT)
    3940                 :          0 :                 drhs = build_debug_ref_for_model (sad->loc, lacc->base,
    3941                 :            :                                                   lacc->offset, lacc);
    3942                 :          2 :               else if (sad->refreshed == SRA_UDH_RIGHT)
    3943                 :          2 :                 drhs = build_debug_ref_for_model (sad->loc, sad->top_racc->base,
    3944                 :            :                                                   offset, lacc);
    3945                 :            :               else
    3946                 :            :                 drhs = NULL_TREE;
    3947                 :      24692 :               if (drhs
    3948                 :      49384 :                   && !useless_type_conversion_p (lacc->type, TREE_TYPE (drhs)))
    3949                 :        347 :                 drhs = fold_build1_loc (sad->loc, VIEW_CONVERT_EXPR,
    3950                 :            :                                         lacc->type, drhs);
    3951                 :      25639 :               ds = gimple_build_debug_bind (get_access_replacement (lacc),
    3952                 :            :                                             drhs, gsi_stmt (sad->old_gsi));
    3953                 :      25639 :               gsi_insert_after (sad->new_gsi, ds, GSI_NEW_STMT);
    3954                 :            :             }
    3955                 :            :         }
    3956                 :            : 
    3957                 :    1225310 :       if (lacc->first_child)
    3958                 :      35451 :         load_assign_lhs_subreplacements (lacc, sad);
    3959                 :            :     }
    3960                 :     545496 : }
    3961                 :            : 
    3962                 :            : /* Result code for SRA assignment modification.  */
    3963                 :            : enum assignment_mod_result { SRA_AM_NONE,       /* nothing done for the stmt */
    3964                 :            :                              SRA_AM_MODIFIED,  /* stmt changed but not
    3965                 :            :                                                   removed */
    3966                 :            :                              SRA_AM_REMOVED };  /* stmt eliminated */
    3967                 :            : 
    3968                 :            : /* Modify assignments with a CONSTRUCTOR on their RHS.  STMT contains a pointer
    3969                 :            :    to the assignment and GSI is the statement iterator pointing at it.  Returns
    3970                 :            :    the same values as sra_modify_assign.  */
    3971                 :            : 
    3972                 :            : static enum assignment_mod_result
    3973                 :    1658650 : sra_modify_constructor_assign (gimple *stmt, gimple_stmt_iterator *gsi)
    3974                 :            : {
    3975                 :    1658650 :   tree lhs = gimple_assign_lhs (stmt);
    3976                 :    1658650 :   struct access *acc = get_access_for_expr (lhs);
    3977                 :    1658650 :   if (!acc)
    3978                 :            :     return SRA_AM_NONE;
    3979                 :     725078 :   location_t loc = gimple_location (stmt);
    3980                 :            : 
    3981                 :     725078 :   if (gimple_clobber_p (stmt))
    3982                 :            :     {
    3983                 :            :       /* Clobber the replacement variable.  */
    3984                 :     622910 :       clobber_subtree (acc, gsi, !acc->grp_covered, loc);
    3985                 :            :       /* Remove clobbers of fully scalarized variables, they are dead.  */
    3986                 :     622910 :       if (acc->grp_covered)
    3987                 :            :         {
    3988                 :     529002 :           unlink_stmt_vdef (stmt);
    3989                 :     529002 :           gsi_remove (gsi, true);
    3990                 :     529002 :           release_defs (stmt);
    3991                 :     529002 :           return SRA_AM_REMOVED;
    3992                 :            :         }
    3993                 :            :       else
    3994                 :            :         return SRA_AM_MODIFIED;
    3995                 :            :     }
    3996                 :            : 
    3997                 :     102168 :   if (CONSTRUCTOR_NELTS (gimple_assign_rhs1 (stmt)) > 0)
    3998                 :            :     {
    3999                 :            :       /* I have never seen this code path trigger but if it can happen the
    4000                 :            :          following should handle it gracefully.  */
    4001                 :          0 :       if (access_has_children_p (acc))
    4002                 :          0 :         generate_subtree_copies (acc->first_child, lhs, acc->offset, 0, 0, gsi,
    4003                 :            :                                  true, true, loc);
    4004                 :          0 :       return SRA_AM_MODIFIED;
    4005                 :            :     }
    4006                 :            : 
    4007                 :     102168 :   if (acc->grp_covered)
    4008                 :            :     {
    4009                 :      75515 :       init_subtree_with_zero (acc, gsi, false, loc);
    4010                 :      75515 :       unlink_stmt_vdef (stmt);
    4011                 :      75515 :       gsi_remove (gsi, true);
    4012                 :      75515 :       release_defs (stmt);
    4013                 :      75515 :       return SRA_AM_REMOVED;
    4014                 :            :     }
    4015                 :            :   else
    4016                 :            :     {
    4017                 :      26653 :       init_subtree_with_zero (acc, gsi, true, loc);
    4018                 :      26653 :       return SRA_AM_MODIFIED;
    4019                 :            :     }
    4020                 :            : }
    4021                 :            : 
    4022                 :            : /* Create and return a new suitable default definition SSA_NAME for RACC which
    4023                 :            :    is an access describing an uninitialized part of an aggregate that is being
    4024                 :            :    loaded.  REG_TREE is used instead of the actual RACC type if that is not of
    4025                 :            :    a gimple register type.  */
    4026                 :            : 
    4027                 :            : static tree
    4028                 :        432 : get_repl_default_def_ssa_name (struct access *racc, tree reg_type)
    4029                 :            : {
    4030                 :        432 :   gcc_checking_assert (!racc->grp_to_be_replaced
    4031                 :            :                        && !racc->grp_to_be_debug_replaced);
    4032                 :        432 :   if (!racc->replacement_decl)
    4033                 :        432 :     racc->replacement_decl = create_access_replacement (racc, reg_type);
    4034                 :        432 :   return get_or_create_ssa_default_def (cfun, racc->replacement_decl);
    4035                 :            : }
    4036                 :            : 
    4037                 :            : /* Examine both sides of the assignment statement pointed to by STMT, replace
    4038                 :            :    them with a scalare replacement if there is one and generate copying of
    4039                 :            :    replacements if scalarized aggregates have been used in the assignment.  GSI
    4040                 :            :    is used to hold generated statements for type conversions and subtree
    4041                 :            :    copying.  */
    4042                 :            : 
    4043                 :            : static enum assignment_mod_result
    4044                 :   15433300 : sra_modify_assign (gimple *stmt, gimple_stmt_iterator *gsi)
    4045                 :            : {
    4046                 :   15433300 :   struct access *lacc, *racc;
    4047                 :   15433300 :   tree lhs, rhs;
    4048                 :   15433300 :   bool modify_this_stmt = false;
    4049                 :   15433300 :   bool force_gimple_rhs = false;
    4050                 :   15433300 :   location_t loc;
    4051                 :   15433300 :   gimple_stmt_iterator orig_gsi = *gsi;
    4052                 :            : 
    4053                 :   15433300 :   if (!gimple_assign_single_p (stmt))
    4054                 :            :     return SRA_AM_NONE;
    4055                 :   12435300 :   lhs = gimple_assign_lhs (stmt);
    4056                 :   12435300 :   rhs = gimple_assign_rhs1 (stmt);
    4057                 :            : 
    4058                 :   12435300 :   if (TREE_CODE (rhs) == CONSTRUCTOR)
    4059                 :    1658650 :     return sra_modify_constructor_assign (stmt, gsi);
    4060                 :            : 
    4061                 :   10776600 :   if (TREE_CODE (rhs) == REALPART_EXPR || TREE_CODE (lhs) == REALPART_EXPR
    4062                 :   10773900 :       || TREE_CODE (rhs) == IMAGPART_EXPR || TREE_CODE (lhs) == IMAGPART_EXPR
    4063                 :   10770600 :       || TREE_CODE (rhs) == BIT_FIELD_REF || TREE_CODE (lhs) == BIT_FIELD_REF)
    4064                 :            :     {
    4065                 :      11128 :       modify_this_stmt = sra_modify_expr (gimple_assign_rhs1_ptr (stmt),
    4066                 :            :                                           gsi, false);
    4067                 :      11128 :       modify_this_stmt |= sra_modify_expr (gimple_assign_lhs_ptr (stmt),
    4068                 :            :                                            gsi, true);
    4069                 :      17850 :       return modify_this_stmt ? SRA_AM_MODIFIED : SRA_AM_NONE;
    4070                 :            :     }
    4071                 :            : 
    4072                 :   10765500 :   lacc = get_access_for_expr (lhs);
    4073                 :   10765500 :   racc = get_access_for_expr (rhs);
    4074                 :   10765500 :   if (!lacc && !racc)
    4075                 :            :     return SRA_AM_NONE;
    4076                 :            :   /* Avoid modifying initializations of constant-pool replacements.  */
    4077                 :    4713920 :   if (racc && (racc->replacement_decl == lhs))
    4078                 :            :     return SRA_AM_NONE;
    4079                 :            : 
    4080                 :    4713510 :   loc = gimple_location (stmt);
    4081                 :    4713510 :   if (lacc && lacc->grp_to_be_replaced)
    4082                 :            :     {
    4083                 :    1348500 :       lhs = get_access_replacement (lacc);
    4084                 :    1348500 :       gimple_assign_set_lhs (stmt, lhs);
    4085                 :    1348500 :       modify_this_stmt = true;
    4086                 :    1348500 :       if (lacc->grp_partial_lhs)
    4087                 :            :         force_gimple_rhs = true;
    4088                 :    1348500 :       sra_stats.exprs++;
    4089                 :            :     }
    4090                 :            : 
    4091                 :    4713510 :   if (racc && racc->grp_to_be_replaced)
    4092                 :            :     {
    4093                 :    1943740 :       rhs = get_access_replacement (racc);
    4094                 :    1943740 :       modify_this_stmt = true;
    4095                 :    1943740 :       if (racc->grp_partial_lhs)
    4096                 :        412 :         force_gimple_rhs = true;
    4097                 :    1943740 :       sra_stats.exprs++;
    4098                 :            :     }
    4099                 :    2769770 :   else if (racc
    4100                 :            :            && !racc->grp_unscalarized_data
    4101                 :     958179 :            && !racc->grp_unscalarizable_region
    4102                 :     890927 :            && TREE_CODE (lhs) == SSA_NAME
    4103                 :    2770230 :            && !access_has_replacements_p (racc))
    4104                 :            :     {
    4105                 :        432 :       rhs = get_repl_default_def_ssa_name (racc, TREE_TYPE (lhs));
    4106                 :        432 :       modify_this_stmt = true;
    4107                 :        432 :       sra_stats.exprs++;
    4108                 :            :     }
    4109                 :            : 
    4110                 :    4713510 :   if (modify_this_stmt)
    4111                 :            :     {
    4112                 :    3197320 :       if (!useless_type_conversion_p (TREE_TYPE (lhs), TREE_TYPE (rhs)))
    4113                 :            :         {
    4114                 :            :           /* If we can avoid creating a VIEW_CONVERT_EXPR do so.
    4115                 :            :              ???  This should move to fold_stmt which we simply should
    4116                 :            :              call after building a VIEW_CONVERT_EXPR here.  */
    4117                 :     267180 :           if (AGGREGATE_TYPE_P (TREE_TYPE (lhs))
    4118                 :     194760 :               && !contains_bitfld_component_ref_p (lhs))
    4119                 :            :             {
    4120                 :      61495 :               lhs = build_ref_for_model (loc, lhs, 0, racc, gsi, false);
    4121                 :      61495 :               gimple_assign_set_lhs (stmt, lhs);
    4122                 :            :             }
    4123                 :      72420 :           else if (lacc
    4124                 :      53148 :                    && AGGREGATE_TYPE_P (TREE_TYPE (rhs))
    4125                 :     104262 :                    && !contains_vce_or_bfcref_p (rhs))
    4126                 :      31786 :             rhs = build_ref_for_model (loc, rhs, 0, lacc, gsi, false);
    4127                 :            : 
    4128                 :     133915 :           if (!useless_type_conversion_p (TREE_TYPE (lhs), TREE_TYPE (rhs)))
    4129                 :            :             {
    4130                 :      40634 :               rhs = fold_build1_loc (loc, VIEW_CONVERT_EXPR, TREE_TYPE (lhs),
    4131                 :            :                                      rhs);
    4132                 :      40634 :               if (is_gimple_reg_type (TREE_TYPE (lhs))
    4133                 :      40634 :                   && TREE_CODE (lhs) != SSA_NAME)
    4134                 :            :                 force_gimple_rhs = true;
    4135                 :            :             }
    4136                 :            :         }
    4137                 :            :     }
    4138                 :            : 
    4139                 :    4713510 :   if (lacc && lacc->grp_to_be_debug_replaced)
    4140                 :            :     {
    4141                 :      50652 :       tree dlhs = get_access_replacement (lacc);
    4142                 :      50652 :       tree drhs = unshare_expr (rhs);
    4143                 :      50652 :       if (!useless_type_conversion_p (TREE_TYPE (dlhs), TREE_TYPE (drhs)))
    4144                 :            :         {
    4145                 :         22 :           if (AGGREGATE_TYPE_P (TREE_TYPE (drhs))
    4146                 :         21 :               && !contains_vce_or_bfcref_p (drhs))
    4147                 :         10 :             drhs = build_debug_ref_for_model (loc, drhs, 0, lacc);
    4148                 :         11 :           if (drhs
    4149                 :         22 :               && !useless_type_conversion_p (TREE_TYPE (dlhs),
    4150                 :         11 :                                              TREE_TYPE (drhs)))
    4151                 :          1 :             drhs = fold_build1_loc (loc, VIEW_CONVERT_EXPR,
    4152                 :          1 :                                     TREE_TYPE (dlhs), drhs);
    4153                 :            :         }
    4154                 :      50652 :       gdebug *ds = gimple_build_debug_bind (dlhs, drhs, stmt);
    4155                 :      50652 :       gsi_insert_before (gsi, ds, GSI_SAME_STMT);
    4156                 :            :     }
    4157                 :            : 
    4158                 :            :   /* From this point on, the function deals with assignments in between
    4159                 :            :      aggregates when at least one has scalar reductions of some of its
    4160                 :            :      components.  There are three possible scenarios: Both the LHS and RHS have
    4161                 :            :      to-be-scalarized components, 2) only the RHS has or 3) only the LHS has.
    4162                 :            : 
    4163                 :            :      In the first case, we would like to load the LHS components from RHS
    4164                 :            :      components whenever possible.  If that is not possible, we would like to
    4165                 :            :      read it directly from the RHS (after updating it by storing in it its own
    4166                 :            :      components).  If there are some necessary unscalarized data in the LHS,
    4167                 :            :      those will be loaded by the original assignment too.  If neither of these
    4168                 :            :      cases happen, the original statement can be removed.  Most of this is done
    4169                 :            :      by load_assign_lhs_subreplacements.
    4170                 :            : 
    4171                 :            :      In the second case, we would like to store all RHS scalarized components
    4172                 :            :      directly into LHS and if they cover the aggregate completely, remove the
    4173                 :            :      statement too.  In the third case, we want the LHS components to be loaded
    4174                 :            :      directly from the RHS (DSE will remove the original statement if it
    4175                 :            :      becomes redundant).
    4176                 :            : 
    4177                 :            :      This is a bit complex but manageable when types match and when unions do
    4178                 :            :      not cause confusion in a way that we cannot really load a component of LHS
    4179                 :            :      from the RHS or vice versa (the access representing this level can have
    4180                 :            :      subaccesses that are accessible only through a different union field at a
    4181                 :            :      higher level - different from the one used in the examined expression).
    4182                 :            :      Unions are fun.
    4183                 :            : 
    4184                 :            :      Therefore, I specially handle a fourth case, happening when there is a
    4185                 :            :      specific type cast or it is impossible to locate a scalarized subaccess on
    4186                 :            :      the other side of the expression.  If that happens, I simply "refresh" the
    4187                 :            :      RHS by storing in it is scalarized components leave the original statement
    4188                 :            :      there to do the copying and then load the scalar replacements of the LHS.
    4189                 :            :      This is what the first branch does.  */
    4190                 :            : 
    4191                 :    4713510 :   if (modify_this_stmt
    4192                 :    1516190 :       || gimple_has_volatile_ops (stmt)
    4193                 :    1516030 :       || contains_vce_or_bfcref_p (rhs)
    4194                 :    1373410 :       || contains_vce_or_bfcref_p (lhs)
    4195                 :    6086540 :       || stmt_ends_bb_p (stmt))
    4196                 :            :     {
    4197                 :            :       /* No need to copy into a constant-pool, it comes pre-initialized.  */
    4198                 :    3396060 :       if (access_has_children_p (racc) && !constant_decl_p (racc->base))
    4199                 :       5459 :         generate_subtree_copies (racc->first_child, rhs, racc->offset, 0, 0,
    4200                 :            :                                  gsi, false, false, loc);
    4201                 :    3390600 :       if (access_has_children_p (lacc))
    4202                 :            :         {
    4203                 :     189614 :           gimple_stmt_iterator alt_gsi = gsi_none ();
    4204                 :     189614 :           if (stmt_ends_bb_p (stmt))
    4205                 :            :             {
    4206                 :      49922 :               alt_gsi = gsi_start_edge (single_non_eh_succ (gsi_bb (*gsi)));
    4207                 :      49922 :               gsi = &alt_gsi;
    4208                 :            :             }
    4209                 :     189614 :           generate_subtree_copies (lacc->first_child, lhs, lacc->offset, 0, 0,
    4210                 :            :                                    gsi, true, true, loc);
    4211                 :            :         }
    4212                 :    3390600 :       sra_stats.separate_lhs_rhs_handling++;
    4213                 :            : 
    4214                 :            :       /* This gimplification must be done after generate_subtree_copies,
    4215                 :            :          lest we insert the subtree copies in the middle of the gimplified
    4216                 :            :          sequence.  */
    4217                 :    3390600 :       if (force_gimple_rhs)
    4218                 :      21826 :         rhs = force_gimple_operand_gsi (&orig_gsi, rhs, true, NULL_TREE,
    4219                 :            :                                         true, GSI_SAME_STMT);
    4220                 :    3390600 :       if (gimple_assign_rhs1 (stmt) != rhs)
    4221                 :            :         {
    4222                 :    1997280 :           modify_this_stmt = true;
    4223                 :    1997280 :           gimple_assign_set_rhs_from_tree (&orig_gsi, rhs);
    4224                 :    1997280 :           gcc_assert (stmt == gsi_stmt (orig_gsi));
    4225                 :            :         }
    4226                 :            : 
    4227                 :    3583880 :       return modify_this_stmt ? SRA_AM_MODIFIED : SRA_AM_NONE;
    4228                 :            :     }
    4229                 :            :   else
    4230                 :            :     {
    4231                 :    2017320 :       if (access_has_children_p (lacc)
    4232                 :    1322920 :           && access_has_children_p (racc)
    4233                 :            :           /* When an access represents an unscalarizable region, it usually
    4234                 :            :              represents accesses with variable offset and thus must not be used
    4235                 :            :              to generate new memory accesses.  */
    4236                 :     510055 :           && !lacc->grp_unscalarizable_region
    4237                 :     510051 :           && !racc->grp_unscalarizable_region)
    4238                 :            :         {
    4239                 :     510045 :           struct subreplacement_assignment_data sad;
    4240                 :            : 
    4241                 :     510045 :           sad.left_offset = lacc->offset;
    4242                 :     510045 :           sad.assignment_lhs = lhs;
    4243                 :     510045 :           sad.assignment_rhs = rhs;
    4244                 :     510045 :           sad.top_racc = racc;
    4245                 :     510045 :           sad.old_gsi = *gsi;
    4246                 :     510045 :           sad.new_gsi = gsi;
    4247                 :     510045 :           sad.loc = gimple_location (stmt);
    4248                 :     510045 :           sad.refreshed = SRA_UDH_NONE;
    4249                 :            : 
    4250                 :     510045 :           if (lacc->grp_read && !lacc->grp_covered)
    4251                 :      34729 :             handle_unscalarized_data_in_subtree (&sad);
    4252                 :            : 
    4253                 :     510045 :           load_assign_lhs_subreplacements (lacc, &sad);
    4254                 :     510045 :           if (sad.refreshed != SRA_UDH_RIGHT)
    4255                 :            :             {
    4256                 :     497633 :               gsi_next (gsi);
    4257                 :     497633 :               unlink_stmt_vdef (stmt);
    4258                 :     497633 :               gsi_remove (&sad.old_gsi, true);
    4259                 :     497633 :               release_defs (stmt);
    4260                 :     497633 :               sra_stats.deleted++;
    4261                 :     497633 :               return SRA_AM_REMOVED;
    4262                 :            :             }
    4263                 :            :         }
    4264                 :            :       else
    4265                 :            :         {
    4266                 :     812864 :           if (access_has_children_p (racc)
    4267                 :     401290 :               && !racc->grp_unscalarized_data
    4268                 :     388069 :               && TREE_CODE (lhs) != SSA_NAME)
    4269                 :            :             {
    4270                 :     388058 :               if (dump_file)
    4271                 :            :                 {
    4272                 :          1 :                   fprintf (dump_file, "Removing load: ");
    4273                 :          1 :                   print_gimple_stmt (dump_file, stmt, 0);
    4274                 :            :                 }
    4275                 :     388058 :               generate_subtree_copies (racc->first_child, lhs,
    4276                 :            :                                        racc->offset, 0, 0, gsi,
    4277                 :            :                                        false, false, loc);
    4278                 :     388058 :               gcc_assert (stmt == gsi_stmt (*gsi));
    4279                 :     388058 :               unlink_stmt_vdef (stmt);
    4280                 :     388058 :               gsi_remove (gsi, true);
    4281                 :     388058 :               release_defs (stmt);
    4282                 :     388058 :               sra_stats.deleted++;
    4283                 :     388058 :               return SRA_AM_REMOVED;
    4284                 :            :             }
    4285                 :            :           /* Restore the aggregate RHS from its components so the
    4286                 :            :              prevailing aggregate copy does the right thing.  */
    4287                 :     424806 :           if (access_has_children_p (racc))
    4288                 :      13232 :             generate_subtree_copies (racc->first_child, rhs, racc->offset, 0, 0,
    4289                 :            :                                      gsi, false, false, loc);
    4290                 :            :           /* Re-load the components of the aggregate copy destination.
    4291                 :            :              But use the RHS aggregate to load from to expose more
    4292                 :            :              optimization opportunities.  */
    4293                 :     424806 :           if (access_has_children_p (lacc))
    4294                 :     184359 :             generate_subtree_copies (lacc->first_child, rhs, lacc->offset,
    4295                 :            :                                      0, 0, gsi, true, true, loc);
    4296                 :            :         }
    4297                 :            : 
    4298                 :     437218 :       return SRA_AM_NONE;
    4299                 :            :     }
    4300                 :            : }
    4301                 :            : 
    4302                 :            : /* Set any scalar replacements of values in the constant pool to the initial
    4303                 :            :    value of the constant.  (Constant-pool decls like *.LC0 have effectively
    4304                 :            :    been initialized before the program starts, we must do the same for their
    4305                 :            :    replacements.)  Thus, we output statements like 'SR.1 = *.LC0[0];' into
    4306                 :            :    the function's entry block.  */
    4307                 :            : 
    4308                 :            : static void
    4309                 :     261601 : initialize_constant_pool_replacements (void)
    4310                 :            : {
    4311                 :     261601 :   gimple_seq seq = NULL;
    4312                 :     261601 :   gimple_stmt_iterator gsi = gsi_start (seq);
    4313                 :     261601 :   bitmap_iterator bi;
    4314                 :     261601 :   unsigned i;
    4315                 :            : 
    4316                 :    1659940 :   EXECUTE_IF_SET_IN_BITMAP (candidate_bitmap, 0, i, bi)
    4317                 :            :     {
    4318                 :    1398340 :       tree var = candidate (i);
    4319                 :    1398340 :       if (!constant_decl_p (var))
    4320                 :    1398330 :         continue;
    4321                 :            : 
    4322                 :         13 :       struct access *access = get_first_repr_for_decl (var);
    4323                 :            : 
    4324                 :        536 :       while (access)
    4325                 :            :         {
    4326                 :        523 :           if (access->replacement_decl)
    4327                 :            :             {
    4328                 :        411 :               gassign *stmt
    4329                 :        411 :                 = gimple_build_assign (get_access_replacement (access),
    4330                 :            :                                        unshare_expr (access->expr));
    4331                 :        411 :               if (dump_file && (dump_flags & TDF_DETAILS))
    4332                 :            :                 {
    4333                 :          0 :                   fprintf (dump_file, "Generating constant initializer: ");
    4334                 :          0 :                   print_gimple_stmt (dump_file, stmt, 0);
    4335                 :          0 :                   fprintf (dump_file, "\n");
    4336                 :            :                 }
    4337                 :        411 :               gsi_insert_after (&gsi, stmt, GSI_NEW_STMT);
    4338                 :        411 :               update_stmt (stmt);
    4339                 :            :             }
    4340                 :            : 
    4341                 :        523 :           if (access->first_child)
    4342                 :            :             access = access->first_child;
    4343                 :        411 :           else if (access->next_sibling)
    4344                 :            :             access = access->next_sibling;
    4345                 :            :           else
    4346                 :            :             {
    4347                 :        215 :               while (access->parent && !access->next_sibling)
    4348                 :            :                 access = access->parent;
    4349                 :        103 :               if (access->next_sibling)
    4350                 :            :                 access = access->next_sibling;
    4351                 :            :               else
    4352                 :         13 :                 access = access->next_grp;
    4353                 :            :             }
    4354                 :            :         }
    4355                 :            :     }
    4356                 :            : 
    4357                 :     261601 :   seq = gsi_seq (gsi);
    4358                 :     261601 :   if (seq)
    4359                 :          7 :     gsi_insert_seq_on_edge_immediate (
    4360                 :          7 :       single_succ_edge (ENTRY_BLOCK_PTR_FOR_FN (cfun)), seq);
    4361                 :     261601 : }
    4362                 :            : 
    4363                 :            : /* Traverse the function body and all modifications as decided in
    4364                 :            :    analyze_all_variable_accesses.  Return true iff the CFG has been
    4365                 :            :    changed.  */
    4366                 :            : 
    4367                 :            : static bool
    4368                 :     261601 : sra_modify_function_body (void)
    4369                 :            : {
    4370                 :     261601 :   bool cfg_changed = false;
    4371                 :     261601 :   basic_block bb;
    4372                 :            : 
    4373                 :     261601 :   initialize_constant_pool_replacements ();
    4374                 :            : 
    4375                 :    6282170 :   FOR_EACH_BB_FN (bb, cfun)
    4376                 :            :     {
    4377                 :    6020570 :       gimple_stmt_iterator gsi = gsi_start_bb (bb);
    4378                 :   46332500 :       while (!gsi_end_p (gsi))
    4379                 :            :         {
    4380                 :   40312000 :           gimple *stmt = gsi_stmt (gsi);
    4381                 :   40312000 :           enum assignment_mod_result assign_result;
    4382                 :   40312000 :           bool modified = false, deleted = false;
    4383                 :   40312000 :           tree *t;
    4384                 :   40312000 :           unsigned i;
    4385                 :            : 
    4386                 :   40312000 :           switch (gimple_code (stmt))
    4387                 :            :             {
    4388                 :     260470 :             case GIMPLE_RETURN:
    4389                 :     260470 :               t = gimple_return_retval_ptr (as_a <greturn *> (stmt));
    4390                 :     260470 :               if (*t != NULL_TREE)
    4391                 :     173196 :                 modified |= sra_modify_expr (t, &gsi, false);
    4392                 :            :               break;
    4393                 :            : 
    4394                 :   15433300 :             case GIMPLE_ASSIGN:
    4395                 :   15433300 :               assign_result = sra_modify_assign (stmt, &gsi);
    4396                 :   15433300 :               modified |= assign_result == SRA_AM_MODIFIED;
    4397                 :   15433300 :               deleted = assign_result == SRA_AM_REMOVED;
    4398                 :   15433300 :               break;
    4399                 :            : 
    4400                 :            :             case GIMPLE_CALL:
    4401                 :            :               /* Operands must be processed before the lhs.  */
    4402                 :    7982910 :               for (i = 0; i < gimple_call_num_args (stmt); i++)
    4403                 :            :                 {
    4404                 :    5286380 :                   t = gimple_call_arg_ptr (stmt, i);
    4405                 :    5286380 :                   modified |= sra_modify_expr (t, &gsi, false);
    4406                 :            :                 }
    4407                 :            : 
    4408                 :    2696530 :               if (gimple_call_lhs (stmt))
    4409                 :            :                 {
    4410                 :    1237990 :                   t = gimple_call_lhs_ptr (stmt);
    4411                 :    1237990 :                   modified |= sra_modify_expr (t, &gsi, true);
    4412                 :            :                 }
    4413                 :            :               break;
    4414                 :            : 
    4415                 :       3215 :             case GIMPLE_ASM:
    4416                 :       3215 :               {
    4417                 :       3215 :                 gasm *asm_stmt = as_a <gasm *> (stmt);
    4418                 :       4465 :                 for (i = 0; i < gimple_asm_ninputs (asm_stmt); i++)
    4419                 :            :                   {
    4420                 :       1250 :                     t = &TREE_VALUE (gimple_asm_input_op (asm_stmt, i));
    4421                 :       1250 :                     modified |= sra_modify_expr (t, &gsi, false);
    4422                 :            :                   }
    4423                 :       4064 :                 for (i = 0; i < gimple_asm_noutputs (asm_stmt); i++)
    4424                 :            :                   {
    4425                 :        849 :                     t = &TREE_VALUE (gimple_asm_output_op (asm_stmt, i));
    4426                 :        849 :                     modified |= sra_modify_expr (t, &gsi, true);
    4427                 :            :                   }
    4428                 :            :               }
    4429                 :            :               break;
    4430                 :            : 
    4431                 :            :             default:
    4432                 :            :               break;
    4433                 :            :             }
    4434                 :            : 
    4435                 :   18306200 :           if (modified)
    4436                 :            :             {
    4437                 :    3421330 :               update_stmt (stmt);
    4438                 :    3421330 :               if (maybe_clean_eh_stmt (stmt)
    4439                 :    3421330 :                   && gimple_purge_dead_eh_edges (gimple_bb (stmt)))
    4440                 :            :                 cfg_changed = true;
    4441                 :            :             }
    4442                 :   40312000 :           if (!deleted)
    4443                 :   38821800 :             gsi_next (&gsi);
    4444                 :            :         }
    4445                 :            :     }
    4446                 :            : 
    4447                 :     261601 :   gsi_commit_edge_inserts ();
    4448                 :     261601 :   return cfg_changed;
    4449                 :            : }
    4450                 :            : 
    4451                 :            : /* Generate statements initializing scalar replacements of parts of function
    4452                 :            :    parameters.  */
    4453                 :            : 
    4454                 :            : static void
    4455                 :     261601 : initialize_parameter_reductions (void)
    4456                 :            : {
    4457                 :     261601 :   gimple_stmt_iterator gsi;
    4458                 :     261601 :   gimple_seq seq = NULL;
    4459                 :     261601 :   tree parm;
    4460                 :            : 
    4461                 :     261601 :   gsi = gsi_start (seq);
    4462                 :     737760 :   for (parm = DECL_ARGUMENTS (current_function_decl);
    4463                 :     737760 :        parm;
    4464                 :     476159 :        parm = DECL_CHAIN (parm))
    4465                 :            :     {
    4466                 :     476159 :       vec<access_p> *access_vec;
    4467                 :     476159 :       struct access *access;
    4468                 :            : 
    4469                 :     476159 :       if (!bitmap_bit_p (candidate_bitmap, DECL_UID (parm)))
    4470                 :     436238 :         continue;
    4471                 :      39921 :       access_vec = get_base_access_vector (parm);
    4472                 :      39921 :       if (!access_vec)
    4473                 :          0 :         continue;
    4474                 :            : 
    4475                 :      39921 :       for (access = (*access_vec)[0];
    4476                 :      97965 :            access;
    4477                 :      58044 :            access = access->next_grp)
    4478                 :      58044 :         generate_subtree_copies (access, parm, 0, 0, 0, &gsi, true, true,
    4479                 :      58044 :                                  EXPR_LOCATION (parm));
    4480                 :            :     }
    4481                 :            : 
    4482                 :     261601 :   seq = gsi_seq (gsi);
    4483                 :     261601 :   if (seq)
    4484                 :      32268 :     gsi_insert_seq_on_edge_immediate (single_succ_edge (ENTRY_BLOCK_PTR_FOR_FN (cfun)), seq);
    4485                 :     261601 : }
    4486                 :            : 
    4487                 :            : /* The "main" function of intraprocedural SRA passes.  Runs the analysis and if
    4488                 :            :    it reveals there are components of some aggregates to be scalarized, it runs
    4489                 :            :    the required transformations.  */
    4490                 :            : static unsigned int
    4491                 :    2266800 : perform_intra_sra (void)
    4492                 :            : {
    4493                 :    2266800 :   int ret = 0;
    4494                 :    2266800 :   sra_initialize ();
    4495                 :            : 
    4496                 :    2266800 :   if (!find_var_candidates ())
    4497                 :    1774510 :     goto out;
    4498                 :            : 
    4499                 :     492288 :   if (!scan_function ())
    4500                 :      44011 :     goto out;
    4501                 :            : 
    4502                 :     448277 :   if (!analyze_all_variable_accesses ())
    4503                 :     186676 :     goto out;
    4504                 :            : 
    4505                 :     261601 :   if (sra_modify_function_body ())
    4506                 :            :     ret = TODO_update_ssa | TODO_cleanup_cfg;
    4507                 :            :   else
    4508                 :     261601 :     ret = TODO_update_ssa;
    4509                 :     261601 :   initialize_parameter_reductions ();
    4510                 :            : 
    4511                 :     261601 :   statistics_counter_event (cfun, "Scalar replacements created",
    4512                 :            :                             sra_stats.replacements);
    4513                 :     261601 :   statistics_counter_event (cfun, "Modified expressions", sra_stats.exprs);
    4514                 :     261601 :   statistics_counter_event (cfun, "Subtree copy stmts",
    4515                 :            :                             sra_stats.subtree_copies);
    4516                 :     261601 :   statistics_counter_event (cfun, "Subreplacement stmts",
    4517                 :            :                             sra_stats.subreplacements);
    4518                 :     261601 :   statistics_counter_event (cfun, "Deleted stmts", sra_stats.deleted);
    4519                 :     261601 :   statistics_counter_event (cfun, "Separate LHS and RHS handling",
    4520                 :            :                             sra_stats.separate_lhs_rhs_handling);
    4521                 :            : 
    4522                 :    2266800 :  out:
    4523                 :    2266800 :   sra_deinitialize ();
    4524                 :    2266800 :   return ret;
    4525                 :            : }
    4526                 :            : 
    4527                 :            : /* Perform early intraprocedural SRA.  */
    4528                 :            : static unsigned int
    4529                 :    1581440 : early_intra_sra (void)
    4530                 :            : {
    4531                 :    1581440 :   sra_mode = SRA_MODE_EARLY_INTRA;
    4532                 :          0 :   return perform_intra_sra ();
    4533                 :            : }
    4534                 :            : 
    4535                 :            : /* Perform "late" intraprocedural SRA.  */
    4536                 :            : static unsigned int
    4537                 :     685350 : late_intra_sra (void)
    4538                 :            : {
    4539                 :     685350 :   sra_mode = SRA_MODE_INTRA;
    4540                 :          0 :   return perform_intra_sra ();
    4541                 :            : }
    4542                 :            : 
    4543                 :            : 
    4544                 :            : static bool
    4545                 :    2269790 : gate_intra_sra (void)
    4546                 :            : {
    4547                 :    2266910 :   return flag_tree_sra != 0 && dbg_cnt (tree_sra);
    4548                 :            : }
    4549                 :            : 
    4550                 :            : 
    4551                 :            : namespace {
    4552                 :            : 
    4553                 :            : const pass_data pass_data_sra_early =
    4554                 :            : {
    4555                 :            :   GIMPLE_PASS, /* type */
    4556                 :            :   "esra", /* name */
    4557                 :            :   OPTGROUP_NONE, /* optinfo_flags */
    4558                 :            :   TV_TREE_SRA, /* tv_id */
    4559                 :            :   ( PROP_cfg | PROP_ssa ), /* properties_required */
    4560                 :            :   0, /* properties_provided */
    4561                 :            :   0, /* properties_destroyed */
    4562                 :            :   0, /* todo_flags_start */
    4563                 :            :   TODO_update_ssa, /* todo_flags_finish */
    4564                 :            : };
    4565                 :            : 
    4566                 :            : class pass_sra_early : public gimple_opt_pass
    4567                 :            : {
    4568                 :            : public:
    4569                 :     200540 :   pass_sra_early (gcc::context *ctxt)
    4570                 :     401080 :     : gimple_opt_pass (pass_data_sra_early, ctxt)
    4571                 :            :   {}
    4572                 :            : 
    4573                 :            :   /* opt_pass methods: */
    4574                 :    1584130 :   virtual bool gate (function *) { return gate_intra_sra (); }
    4575                 :    1581440 :   virtual unsigned int execute (function *) { return early_intra_sra (); }
    4576                 :            : 
    4577                 :            : }; // class pass_sra_early
    4578                 :            : 
    4579                 :            : } // anon namespace
    4580                 :            : 
    4581                 :            : gimple_opt_pass *
    4582                 :     200540 : make_pass_sra_early (gcc::context *ctxt)
    4583                 :            : {
    4584                 :     200540 :   return new pass_sra_early (ctxt);
    4585                 :            : }
    4586                 :            : 
    4587                 :            : namespace {
    4588                 :            : 
    4589                 :            : const pass_data pass_data_sra =
    4590                 :            : {
    4591                 :            :   GIMPLE_PASS, /* type */
    4592                 :            :   "sra", /* name */
    4593                 :            :   OPTGROUP_NONE, /* optinfo_flags */
    4594                 :            :   TV_TREE_SRA, /* tv_id */
    4595                 :            :   ( PROP_cfg | PROP_ssa ), /* properties_required */
    4596                 :            :   0, /* properties_provided */
    4597                 :            :   0, /* properties_destroyed */
    4598                 :            :   TODO_update_address_taken, /* todo_flags_start */
    4599                 :            :   TODO_update_ssa, /* todo_flags_finish */
    4600                 :            : };
    4601                 :            : 
    4602                 :            : class pass_sra : public gimple_opt_pass
    4603                 :            : {
    4604                 :            : public:
    4605                 :     200540 :   pass_sra (gcc::context *ctxt)
    4606                 :     401080 :     : gimple_opt_pass (pass_data_sra, ctxt)
    4607                 :            :   {}
    4608                 :            : 
    4609                 :            :   /* opt_pass methods: */
    4610                 :     685664 :   virtual bool gate (function *) { return gate_intra_sra (); }
    4611                 :     685350 :   virtual unsigned int execute (function *) { return late_intra_sra (); }
    4612                 :            : 
    4613                 :            : }; // class pass_sra
    4614                 :            : 
    4615                 :            : } // anon namespace
    4616                 :            : 
    4617                 :            : gimple_opt_pass *
    4618                 :     200540 : make_pass_sra (gcc::context *ctxt)
    4619                 :            : {
    4620                 :     200540 :   return new pass_sra (ctxt);
    4621                 :            : }

Generated by: LCOV version 1.0

LCOV profile is generated on x86_64 machine using following configure options: configure --disable-bootstrap --enable-coverage=opt --enable-languages=c,c++,fortran,go,jit,lto --enable-host-shared. GCC test suite is run with the built compiler.