SOLR-13681: make Lucene's index sorting directly configurable in Solr by cpoerschke · Pull Request #313 · apache/solr

cpoerschke · 2021-09-28T13:15:23Z

https://issues.apache.org/jira/browse/SOLR-13681

see also https://issues.apache.org/jira/browse/SOLR-17310 scribbles w.r.t. potentially combining "within segment sorting" (e.g. this PR) and "between segments sorting" (e.g. @weiwang19's #2477 PR) in one configuration element

Resolved Conflicts: solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc

janhoy

Great.

Add a CHANGES entry. I may try to re-format the RefGuide edits from the patch later...

janhoy · 2021-09-28T18:38:47Z

    final Sort expected = new Sort(new SortField(expectedFieldName, expectedFieldType, expectedFieldSortDescending));
    final Sort actual = sortingMergePolicy.getSort();
    assertEquals("SortingMergePolicy.getSort", expected, actual);
+    assertEquals("indexSort", expected, iwc.getIndexSort());


Let's create a new test method for the new config option, which will survive removal of SMP. Instead of creating yet another solrconfig-xyz.xml I think it should be possible to let the test invoke config-api to set the new setting.

Yes, I would also favour not creating yet another solrconfig-xyz.xml file. Have used the config-api in tests before but from my research done so far it seems indexConfig is not yet supported by it, or rather 'get' works but 'set' does not?

Also interestingly V2 'get' works but V1 does not, for indexConfig but not for (say) requestHandler

both work

curl "http://localhost:8983/solr/gettingstarted/config/requestHandler" curl "http://localhost:8983/api/collections/gettingstarted/config/requestHandler"

latter works, former does not

curl "http://localhost:8983/solr/gettingstarted/config/indexConfig" curl "http://localhost:8983/api/collections/gettingstarted/config/indexConfig"

https://solr.apache.org/guide/8_10/config-api.html#retrieving-the-config lists indexConfig as available top-level section.

$ curl "http://localhost:8983/solr/gettingstarted/config/indexConfig" <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404 Not Found</h2> <table> <tr><th>URI:</th><td>/solr/gettingstarted/config/indexConfig</td></tr> <tr><th>STATUS:</th><td>404</td></tr> <tr><th>MESSAGE:</th><td>Not Found</td></tr> <tr><th>SERVLET:</th><td>default</td></tr> </table> </body> </html>

Ok, perhaps they are R/O because it is not supported to change these after collection creation?
In that case, a new xml file is probably better anyway.

Are you planning to validate that index sorting is working in the test, or simply that the config is set? I think perhaps a new method in the existing IndexConfig test would be enough?

Yes, and perhaps changing some indexConfig parameters after collection creation would be fine but not so for others. Also the merge policy (factory) configuration is nested, not sure if config-api would easily support that.

Added a new method, modelled on the existing sorting-merge-policy one that will go away with sorting merge policy. There is existing TestSegmentSorting coverage w.r.t. index sorting working in a test, so it might make sense to tap into that somehow?

This reverts commit 8171b27.

…ter-part to testSortingMPSolrIndexConfigCreation

…ig-indexSort.xml dual test coverage (latter still failing)

This reverts commit 7514d34.

cpoerschke · 2021-09-29T12:39:19Z

    if (cmd.getSegmentTerminateEarly()) {
      final Sort cmdSort = cmd.getSort();
      final int cmdLen = cmd.getLen();
-      final Sort mergeSort = core.getSolrCoreState().getMergePolicySort();
+      final Sort mergeSort = core.getSolrCoreState().getMergePolicySort(); // TODO: need this account for indexSort also?


https://issues.apache.org/jira/browse/SOLR-15390 is about deprecating the segmentTerminateEarly flag.

…igured

cpoerschke · 2021-09-29T12:48:21Z

Great.

Add a CHANGES entry. I may try to re-format the RefGuide edits from the patch later...

Thanks for attaching SOLR-13681-refguide-skel.patch to the JIRA ticket!

If you wish, Allow edits and access to secrets by maintainers is enabled for this pull request, feel free to add documentation or code or test change commits directly to the pull request branch here.

…estSegmentSorting pass

janhoy · 2021-10-22T07:25:55Z

Sorry for going silent on this, I'll put it on my radar for next week.

Resolved Conflicts: solr/core/src/java/org/apache/solr/update/SolrIndexConfig.java

…er origin/main merge

Resolved Conflicts: solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java solr/core/src/test/org/apache/solr/cloud/TestSegmentSorting.java solr/core/src/test/org/apache/solr/update/SolrIndexConfigTest.java

atris · 2024-05-09T17:09:50Z

@cpoerschke I am about to pick this up and start a fresh PR since this one has gotten stale. Is it ok if I proceed? I will ensure that the PR is marked for your review and you are credited as co author.

Resolved Conflicts: solr/core/src/test/org/apache/solr/update/SolrIndexConfigTest.java

cpoerschke · 2024-05-10T08:47:56Z

@cpoerschke I am about to pick this up and start a fresh PR since this one has gotten stale. Is it ok if I proceed? I will ensure that the PR is marked for your review and you are credited as co author.

@atris - wonderful, yes please proceed. Also I've just resolved the (trivial) merge conflict on this PR and "allow edits by maintainers" is enabled for the PR i.e. as an alternative to a fresh PR you'd also be welcome to add commits to the PR's branch if you like.

atris · 2024-06-13T14:34:16Z

Still working on this - have a PR in process. Will publish by weekend and tag you for review

HoustonPutman · 2026-03-20T18:58:56Z

@cpoerschke is there anything this needs to get over the line? I can help if so.

cpoerschke · 2026-03-27T19:33:57Z

@cpoerschke is there anything this needs to get over the line? I can help if so.

Thanks @HoustonPutman!

From what I (very vaguely) recall there were details around "what if this is enabled for an existing collection" to be figured out, either programmatically somehow or perhaps via documentation. https://issues.apache.org/jira/browse/SOLR-13681 on 02-Sep-2020 has a cross-reference to https://issues.apache.org/jira/browse/SOLR-12239 around this, though no doubt things might have changed since etc.

If you or anyone else has bandwidth to help here that would be wonderful.

cpoerschke added 3 commits September 28, 2021 14:03

git apply SOLR-13681.patch

b37f45c

Merge remote-tracking branch 'origin/main' into SOLR-13681

564a66d

Resolved Conflicts: solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc

make it compile

bc39779

janhoy reviewed Sep 28, 2021

View reviewed changes

cpoerschke added 7 commits September 29, 2021 09:08

code review feedback: logging tweak

e8ecd26

add (skeleton) SolrIndexConfigIndexSortTest class

7514d34

add indexConfig fragments to TestSolrConfigHandler

8171b27

Revert "add indexConfig fragments to TestSolrConfigHandler"

33a722e

This reverts commit 8171b27.

SolrIndexConfigTest: add testIndexSortSolrIndexConfigCreation as coun…

2247d79

…ter-part to testSortingMPSolrIndexConfigCreation

TestSegmentSorting: solrconfig-sortingmergepolicyfactory.xml/solrconf…

1478be7

…ig-indexSort.xml dual test coverage (latter still failing)

Revert "add (skeleton) SolrIndexConfigIndexSortTest class"

14b8c49

This reverts commit 7514d34.

cpoerschke commented Sep 29, 2021

View reviewed changes

Comment thread solr/core/src/test/org/apache/solr/cloud/TestSegmentSorting.java Outdated

SolrIndexConfigTest: assert that indexSort is not present if not conf…

7feca02

…igured

cpoerschke added 2 commits September 30, 2021 12:50

address TODO in SolrIndexSearcher.buildAndRunCollectorChain to make T…

736a75b

…estSegmentSorting pass

Merge remote-tracking branch 'origin/main' into SOLR-13681

3920768

cpoerschke added 4 commits December 22, 2021 13:29

Merge remote-tracking branch 'origin/main' into SOLR-13681

97b1206

Resolved Conflicts: solr/core/src/java/org/apache/solr/update/SolrIndexConfig.java

adjust SolrIndexConfigTest.testIndexSortSolrIndexConfigCreation() aft…

f0889c0

…er origin/main merge

Merge remote-tracking branch 'origin/main' into SOLR-13681

0fd2c9f

Resolved Conflicts: solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java solr/core/src/test/org/apache/solr/cloud/TestSegmentSorting.java solr/core/src/test/org/apache/solr/update/SolrIndexConfigTest.java

./gradlew spotlessApply

eae8527

janhoy requested a review from atris May 7, 2024 12:14

Merge remote-tracking branch 'origin/main' into SOLR-13681

94190d5

Resolved Conflicts: solr/core/src/test/org/apache/solr/update/SolrIndexConfigTest.java

github-actions Bot added tests cat:search cat:cloud cat:index labels May 10, 2024

Merge remote-tracking branch 'origin/main' into SOLR-13681

eee971c

epugh mentioned this pull request Feb 2, 2025

SOLR-17310: Configurable LeafSorter to customize segment search order #2477

Closed

7 tasks

cpoerschke added 2 commits October 17, 2025 14:47

Merge branch 'main' into SOLR-13681

4245609

Merge branch 'apache:main' into SOLR-13681

634c2c8

Merge branch 'apache:main' into SOLR-13681

3b35338

serhiy-bzhezytskyy mentioned this pull request Jul 16, 2026

SOLR-12239: add RESORTINDEX core-admin action to re-sort an existing index #4644

Open

Uh oh!

Conversation

cpoerschke commented Sep 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janhoy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

janhoy Sep 28, 2021

Choose a reason for hiding this comment

Uh oh!

cpoerschke Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

cpoerschke Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

janhoy Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

cpoerschke Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

cpoerschke Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cpoerschke commented Sep 29, 2021

Uh oh!

janhoy commented Oct 22, 2021

Uh oh!

atris commented May 9, 2024

Uh oh!

cpoerschke commented May 10, 2024

Uh oh!

atris commented Jun 13, 2024

Uh oh!

HoustonPutman commented Mar 20, 2026

Uh oh!

cpoerschke commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cpoerschke commented Sep 28, 2021 •

edited

Loading