Skip to content

feat(fts): add code tokenizer with configurable child tokenizer - #772

Open
egolearner wants to merge 1 commit into
alibaba:mainfrom
egolearner:feature/code-tokenizer
Open

egolearner wants to merge 1 commit into
alibaba:mainfrom
egolearner:feature/code-tokenizer

Conversation

@egolearner

Copy link
Copy Markdown
Collaborator

Summary

Add an opt-in code tokenizer for identifier-aware full-text search. getRequestTime emits getrequesttime, get, request, and time, allowing keyword queries to retrieve camelCase and snake_case names. Remaining text uses a configurable child tokenizer: standard by default or jieba with nested parameters. All output is lowercased without requiring a lowercase filter.

Preserve existing flat AND/OR, BM25, and phrase execution semantics. Phrases operate on the expanded token sequence. Offset and token-count overflow guards log an error and return the tokens generated so far instead of throwing. Changing tokenizer configuration requires rebuilding the FTS index.

Validation

  • Built code_tokenizer_test, fts_parser_test, and fts_query_test on macOS.
  • Updated tests cover identifier splitting, Unicode boundaries, child configuration, intrinsic lowercasing, offsets, phrase parsing/retrieval, and FTS index lifecycle.
  • Current test execution is blocked locally: all three binaries were terminated with SIGKILL before producing test output. Runtime results remain unverified for this revision.
  • git diff --check passed. Linux and Windows execution remains pending.

@egolearner
egolearner force-pushed the feature/code-tokenizer branch from 9d11ec2 to 8396136 Compare September 21, 2026 03:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant