Skip to content

ENH: Use array indexing preparation routines for flatiter objects #28590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

ENH: Use array indexing preparation routines for flatiter objects #28590

wants to merge 10 commits into from
+499 −353

Conversation

lysnikolaou
Copy link
Member

  • Use prepare_index in iter_subscript and iter_ass_subscript. This fixes various cases that were broken before:
    • arr.flat[[True, True]]
    • arr.flat[[1.0, 1.0]]
    • arr.flat[()] = 0
  • Add more extensive tests for flatiter indexing operations

Closes #28314.

@lysnikolaou lysnikolaou changed the title [ENH] Use array indexing preparation routines for flatiter objects ENH: Use array indexing preparation routines for flatiter objects Mar 26, 2025
@lysnikolaou lysnikolaou force-pushed the use-prepare-index-flatiter branch from 198df6b to 75aaed0 Compare March 26, 2025 10:23
@lysnikolaou lysnikolaou force-pushed the use-prepare-index-flatiter branch from 75aaed0 to 9f2d51f Compare March 26, 2025 10:30
ngoldbaum
ngoldbaum reviewed Mar 26, 2025
View reviewed changes
numpy/_core/tests/test_regression.py
assert_raises(ValueError, ia, x.flat, s, np.zeros(9, dtype=float))
assert_raises(ValueError, ia, x.flat, s, np.zeros(11, dtype=float))
assert_raises(IndexError, ia, x.flat, s, np.zeros(9, dtype=float))
assert_raises(IndexError, ia, x.flat, s, np.zeros(11, dtype=float))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this is certainly more consistent and I'd even call it a bugfix, it is a behavior change and someone might have code relying on the old behavior. Needs a release note at least. You also need another release note for the new features.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Added a release note that lists all the most important changes.

@lysnikolaou lysnikolaou force-pushed the use-prepare-index-flatiter branch from 8f2b322 to 33109dd Compare March 28, 2025 13:52
@ngoldbaum
Copy link
Member

This is a big refactor, so I think we'll need at least two experienced developers to go over the C code changes, so that might take a while. I'll try to do a pass focusing on the correctness of the C code soon. On a first, high-level pass this looks like mostly simplification and cleanup.

I think you should also try running the indexing benchmarks to see if there are any significant regressions in existing benchmarks. I think bench_indexing already captures several workflows that go through the changed low-level C code path.

It would also be nice to get new entries in the FlatIterIndexing benchmark for newly added functionality.

@ngoldbaum ngoldbaum self-assigned this Mar 28, 2025
@lysnikolaou
Copy link
Member Author

lysnikolaou commented Mar 28, 2025 •
edited
Loading

These are the results of running the (old & new) benchmarks:

| Change   | Before [93898621] <main>   | After [cfcdabf0] <use-prepare-index-flatiter>   |     Ratio | Benchmark (Parameter)                                       |
|----------|----------------------------|-------------------------------------------------|-----------|-------------------------------------------------------------|
| +        | 87.0±0.4ns                 | 43.0±0.2ms                                      | 494276    | bench_indexing.FlatIterIndexing.time_flat_empty_tuple_index |
| +        | 115±3ns                    | 479±8ns                                         |      4.15 | bench_indexing.FlatIterIndexing.time_flat_bool_index_0d     |
| +        | 39.4±0.3ms                 | 42.8±0.3ms                                      |      1.09 | bench_indexing.FlatIterIndexing.time_flat_ellipsis_index    |
| +        | 3.95±0.04ms                | 4.29±0.07ms                                     |      1.09 | bench_indexing.FlatIterIndexing.time_flat_slice_index       |

It looks like having special cases for tuple, ellipses etc. (instead of going through prepare_index) did have an impact on performance. Should we try and keep those special cases in?

@ngoldbaum
Copy link
Member

Should we try and keep those special cases in?

Probably

@lysnikolaou
Copy link
Member Author

Should we try and keep those special cases in?

Probably

I added a couple of special cases for an empty tuple and boolean indexes. This fixes the two worst performance regressions. I feel that the rest are acceptable, since this goes through a much more complex code path to make sure that everything is set up correctly.

@ngoldbaum ngoldbaum added this to the 2.3.0 release milestone Apr 23, 2025
@ngoldbaum
Copy link
Member

I added the 2.3 milestone to make sure we don't drop reviewing this before cutting the release.

@charris
Copy link
Member

charris commented May 19, 2025

I added the 2.3 milestone

@ngoldbaum I am about to push this off to 2.4 unless you want to put it in very soon.

@ngoldbaum
Copy link
Member

I spoke with Lysandros and he said it's OK to push this off. We'll coordinate on getting this reviewed soon.

@charris charris modified the milestones: 2.3.0 release, 2.4.0 release May 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ngoldbaum ngoldbaum ngoldbaum left review comments

Assignees

@ngoldbaum ngoldbaum

Labels
None yet
Projects
None yet
Milestone
2.4.0 release
Development

Successfully merging this pull request may close these issues.

BUG: Fix flatiter indexing
3 participants
@lysnikolaou @ngoldbaum @charris
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy