github-mirror/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2024-12-23 01:50:24 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
13fbab5cf4
gitea/tests/gitea-repositories-meta/org42/search-by-path.git/info/refs

14 lines
806 B
Plaintext
Raw Normal View History

Allow code search by filename (#32210) This is a large and complex PR, so let me explain in detail its changes. First, I had to create new index mappings for Bleve and ElasticSerach as the current ones do not support search by filename. This requires Gitea to recreate the code search indexes (I do not know if this is a breaking change, but I feel it deserves a heads-up). I've used [this approach](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/analysis-pathhierarchy-tokenizer.html) to model the filename index. It allows us to efficiently search for both the full path and the name of a file. Bleve, however, does not support this out-of-box, so I had to code a brand new [token filter](https://blevesearch.com/docs/Token-Filters/) to generate the search terms. I also did an overhaul in the `indexer_test.go` file. It now asserts the order of the expected results (this is important since matches based on the name of a file are more relevant than those based on its content). I've added new test scenarios that deal with searching by filename. They use a new repo included in the Gitea fixture. The screenshot below depicts how Gitea shows the search results. It shows results based on content in the same way as the current version does. In matches based on the filename, the first seven lines of the file contents are shown (BTW, this is how GitHub does it). ![image](https://github.com/user-attachments/assets/9d938d86-1a8d-4f89-8644-1921a473e858) Resolves #32096 --------- Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
2024-10-12 07:35:04 +08:00
90c1019714259b24fb81711d4416ac0f18667dfa refs/heads/DefaultBranch
985f0301dba5e7b34be866819cd15ad3d8f508ee refs/heads/branch2
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/develop
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/feature/1
78fb907e3a3309eae4fe8fef030874cebbf1cd5e refs/heads/home-md-img-check
Updated tokenizer to better matching when search for code snippets (#32261) This PR improves the accuracy of Gitea's code search. Currently, Gitea does not consider statements such as `onsole.log("hello")` as hits when the user searches for `log`. The culprit is how both ES and Bleve are tokenizing the file contents (in both cases, `console.log` is a whole token). In ES' case, we changed the tokenizer to [simple_pattern_split](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-simplepatternsplit-tokenizer.html#:~:text=The%20simple_pattern_split%20tokenizer%20uses%20a,the%20tokenization%20is%20generally%20faster.). In such a case, tokens are words formed by digits and letters. In Bleve's case, it employs a [letter](https://blevesearch.com/docs/Tokenizers/) tokenizer. Resolves #32220 --------- Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
2024-11-07 04:51:20 +08:00
9f894b61946fd2f7b8b9d8e370e4d62f915522f5 refs/heads/master
Allow code search by filename (#32210) This is a large and complex PR, so let me explain in detail its changes. First, I had to create new index mappings for Bleve and ElasticSerach as the current ones do not support search by filename. This requires Gitea to recreate the code search indexes (I do not know if this is a breaking change, but I feel it deserves a heads-up). I've used [this approach](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/analysis-pathhierarchy-tokenizer.html) to model the filename index. It allows us to efficiently search for both the full path and the name of a file. Bleve, however, does not support this out-of-box, so I had to code a brand new [token filter](https://blevesearch.com/docs/Token-Filters/) to generate the search terms. I also did an overhaul in the `indexer_test.go` file. It now asserts the order of the expected results (this is important since matches based on the name of a file are more relevant than those based on its content). I've added new test scenarios that deal with searching by filename. They use a new repo included in the Gitea fixture. The screenshot below depicts how Gitea shows the search results. It shows results based on content in the same way as the current version does. In matches based on the filename, the first seven lines of the file contents are shown (BTW, this is how GitHub does it). ![image](https://github.com/user-attachments/assets/9d938d86-1a8d-4f89-8644-1921a473e858) Resolves #32096 --------- Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
2024-10-12 07:35:04 +08:00
62fb502a7172d4453f0322a2cc85bddffa57f07a refs/heads/pr-to-update
4649299398e4d39a5c09eb4f534df6f1e1eb87cc refs/heads/sub-home-md-img-check
3fa2f829675543ecfc16b2891aebe8bf0608a8f4 refs/notes/commits
4a357436d925b5c974181ff12a994538ddc5a269 refs/pull/2/head
5f22f7d0d95d614d25a5b68592adb345a4b5c7fd refs/pull/3/head
62fb502a7172d4453f0322a2cc85bddffa57f07a refs/pull/5/head
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/tags/v1.1
Reference in New Issue Copy Permalink