From bbe7b0fdb6a7643cb041cecf14ef2a62955d56ae Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Sat, 15 Aug 2026 09:36:37 -0700 Subject: [PATCH] chore: add changelog and release 2.7.0 Backfill CHANGELOG.md from the git history and the RubyGems release dates, back to 1.3.2. Group 1.0 through 1.3.1 under one note, because those versions all reached RubyGems on 2009-07-25 and the history before 2010 does not map to them. Follow the format solid_objects uses: a `## X.Y.Z - YYYY-MM-DD` heading per release, prose bullets that describe user-visible behavior, and a `**Breaking:**` prefix where a caller must change. Bump the version to 2.7.0. The keywords CLI, MultiIO, and String#stem_to_word_hash are new since 2.6.0 and none of them break an existing caller, so this is a minor release. Point changelog_uri at CHANGELOG.md rather than the releases page, and list README.md and CHANGELOG.md in s.files instead of globbing *.md, which would otherwise ship AGENTS.md and CLAUDE.md to gem users. Rename CLAUDE.md to AGENTS.md and leave an @AGENTS.md pointer, matching solid_objects. Document how to write a changelog entry and how to cut a release. --- AGENTS.md | 117 +++++++++++++++++++++++++++++ CHANGELOG.md | 153 ++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 78 +------------------ Gemfile.lock | 2 +- classifier.gemspec | 5 +- lib/classifier/version.rb | 2 +- 6 files changed, 276 insertions(+), 81 deletions(-) create mode 100644 AGENTS.md create mode 100644 CHANGELOG.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..c59aff76 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,117 @@ +# AGENTS.md + +This file provides guidance to coding agents when working with code in this repository. + +## Project Overview + +Ruby gem providing text classification via two algorithms: +- **Bayes** (`Classifier::Bayes`) - Naive Bayesian classification +- **LSI** (`Classifier::LSI`) - Latent Semantic Indexing for semantic classification, clustering, and search + +## Common Commands + +```bash +# Compile native C extension +bundle exec rake compile + +# Run all tests (compiles first) +bundle exec rake test + +# Run a single test file +ruby -Ilib test/bayes/bayesian_test.rb +ruby -Ilib test/lsi/lsi_test.rb + +# Run tests with pure Ruby (no native extension) +NATIVE_VECTOR=true bundle exec rake test + +# Run benchmarks +bundle exec rake benchmark +bundle exec rake benchmark:compare + +# Interactive console +bundle exec rake console + +# Generate documentation +bundle exec rake doc +``` + +## Changelog + +`CHANGELOG.md` records every user-visible change. Add an entry in the same pull +request that makes the change. A reviewer who finds a behavior change without an +entry asks for one. + +Write entries for the person who installs the gem, not for the person who wrote +the commit: + +- Describe the behavior that ships, not the path taken to reach it. A bug that a + reviewer found and the author fixed before release never existed for a user. +- Put the entry under a `## X.Y.Z - YYYY-MM-DD` heading. Use the released + version number without a `v` prefix, and an ISO date. +- Start a breaking change with `**Breaking:**` and say what a caller must now + do differently. +- Name the public constant, method, or command that changed, and show the + command or call when an example makes it concrete. +- Skip changes a user cannot observe: dependency bumps for development, lint + configuration, CI, and internal refactors. +- Order the entries within a release by how much they matter to a user. + +Choose the version with semantic versioning: a breaking change is a major, a new +feature is a minor, and a fix alone is a patch. + +## Release Workflow + +Update `lib/classifier/version.rb`, `CHANGELOG.md`, and `Gemfile.lock`. Run +`bundle exec rake` and `bundle exec rubocop`. Then commit and merge to `master`. + +Publish by pushing an annotated version tag: + +```bash +git tag -a v2.7.0 -m "Version 2.7.0" +git push origin v2.7.0 +``` + +The `Release` workflow runs CI, builds the gem, publishes it through RubyGems +trusted publishing, and creates the GitHub release. Never run `gem push` from a +workstation. + +## Architecture + +### Core Components + +**Bayesian Classifier** (`lib/classifier/bayes.rb`) +- Train with `train(category, text)` or dynamic methods like `train_spam(text)` +- Classify with `classify(text)` returning the best category +- Uses log probabilities for numerical stability + +**LSI Classifier** (`lib/classifier/lsi.rb`) +- Uses Singular Value Decomposition (SVD) for semantic analysis +- Native C extension for 5-50x faster matrix operations; falls back to pure Ruby +- Key operations: `add_item`, `classify`, `find_related`, `search` +- `auto_rebuild` option controls automatic index rebuilding after changes + +**String Extensions** (`lib/classifier/extensions/word_hash.rb`) +- `word_hash` / `clean_word_hash` - tokenize text to stemmed word frequencies +- `CORPUS_SKIP_WORDS` - stopwords filtered during tokenization +- Uses `fast-stemmer` gem for Porter stemming + +**Vector Extensions** (`lib/classifier/extensions/vector.rb`) +- Pure Ruby SVD implementation (`Matrix#SV_decomp`) - used as fallback +- Vector normalization and magnitude calculations + +### Native C Extension (`ext/classifier/`) + +LSI uses a native C extension for fast linear algebra operations: +- `Classifier::Linalg::Vector` - Vector operations (alloc, normalize, dot product) +- `Classifier::Linalg::Matrix` - Matrix operations (alloc, transpose, multiply) +- Jacobi SVD implementation for singular value decomposition + +Check current backend: `Classifier::LSI.backend` returns `:native` or `:ruby` +Force pure Ruby: `NATIVE_VECTOR=true bundle exec rake test` + +### Content Nodes (`lib/classifier/lsi/content_node.rb`) + +Internal data structure storing: +- `word_hash` - term frequencies +- `raw_vector` / `raw_norm` - initial vector representation +- `lsi_vector` / `lsi_norm` - reduced dimensionality representation after SVD diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..1b5f572f --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,153 @@ +# Changelog + +## 2.7.0 - 2026-08-15 + +- Add a `keywords` executable for TF-IDF keyword extraction. `keywords fit` + builds a vocabulary from files or standard input, `keywords extract` reads a + file, `keywords info` prints model statistics, and a bare text argument + transforms that text. The command prints `term:score` pairs and maps stems + back to whole words, so `keywords "Ruby is elegant"` prints + `elegant:0.61 ruby:0.61`. Options set the model path, the top-N terms, quiet + mode, `--min-df`, `--max-df`, and `--ngram`. Usage errors exit 2 and other + errors exit 1. The gem now installs two executables, `classifier` and + `keywords`. +- Treat each line as a separate document during `keywords fit`. The document + count controls the inverse document frequency, so a file with many lines + contributes many documents. +- Add `Classifier::Streaming::MultiIO`. It reads several IO objects or file + paths as one sequential stream. It opens and closes each path one at a time, + so a corpus larger than the file descriptor limit still fits. +- Add `String#stem_to_word_hash`. It maps each stemmed root to the most + frequent original word in the text. +- Add the `min_df` and `max_df` readers to `Classifier::TFIDF`. +- Accept a `MultiIO` in `TFIDF#fit_from_stream` and `Streaming::LineReader`. + +## 2.6.0 - 2026-06-25 + +- Add a `--search` flag to the command line tool to filter local models. +- Add a model detail view to the command line tool. +- Read model information in advance when the tool lists local models. + +## 2.5.0 - 2026-06-01 + +- Accept keyword arguments in `train_from_stream`. + +## 2.4.0 - 2026-05-19 + +- Make `min_word_length` configurable, so a caller can keep or drop short + words during tokenization. +- Add a Claude Code plugin with a skill and slash commands. + +## 2.3.2 - 2026-01-01 + +- Force UTF-8 encoding on the HTTP response body, so a remote model loads + under any locale. + +## 2.3.1 - 2026-01-01 + +- Force UTF-8 encoding when the locale is not UTF-8. Model data and user input + no longer raise an encoding error. + +## 2.3.0 - 2025-12-31 + +- Add a `classifier` executable with a model registry. The tool trains, + classifies, and manages saved models from the shell. +- Fix the examples and the broken links in the README. +- Add Dependabot for automated dependency updates. + +## 2.2.0 - 2025-12-29 + +- **Breaking:** set `required_ruby_version` to `>= 3.1`. Older Ruby versions + are no longer supported. +- Add a k-Nearest Neighbors classifier. +- Add a Logistic Regression classifier. +- Add a TF-IDF vectorizer. +- Add streaming training and incremental SVD, so a corpus larger than memory + can train a model. +- Add a hash-style API to add items to LSI. +- Add keyword arguments to `Bayes#train` and `Bayes#untrain`. +- Accept an array of categories in the classifier constructor. +- Fix the sentence and paragraph splits in `Summary`. +- Add property-based tests for the probabilistic invariants. + +## 2.1.0 - 2025-12-28 + +- Replace the optional GSL dependency with a bundled C extension for LSI. The + extension has no external dependency and falls back to pure Ruby. +- Add pluggable persistence backends through a storage API. +- Add `save` and `load` methods for classifier persistence. +- Add thread safety to the Bayes and LSI classifiers. +- Expose the LSI tuning parameters, with validation and an introspection API. +- Cache the expensive computations in the Bayes classifier. + +## 2.0.0 - 2025-12-27 + +- **Breaking:** replace the fixed 0.1 constant in the Bayes classifier with + add-one (Laplace) smoothing, where + `P(word|category) = (count + 1) / (total + vocabulary_size)`. The smoothing + now scales with the vocabulary size and applies to seen and unseen words + alike. Classification scores change as a result. +- Fix an LSI dimension mismatch in the pure Ruby SVD. +- Fix the numerical stability of the SVD implementation. +- Replace the separate RBS files with inline annotations. +- Add a GitHub Actions workflow that publishes the gem on a version tag. +- Add an LSI benchmark that compares GSL against pure Ruby. +- Add RuboCop and SimpleCov. + +## 1.4.4 - 2024-07-31 + +- Improve the scaling of the LSI content node. + +## 1.4.3 - 2024-07-31 + +- Require `set` and use the explicit `::Set` namespace. +- Refactor `prepare_category_name`. + +## 1.4.2 - 2024-07-31 + +- Fix the word count when `remove_category` runs. +- Add the `mutex_m` dependency and update the `fast-stemmer` version. + +## 1.4.1 - 2024-07-31 + +- Add `remove_category` to the Bayes classifier. + +## 1.4.0 - 2024-07-31 + +- Add `classify_with_confidence` to the LSI classifier. +- Require `mathn` only for Ruby 2.5 and later, and add `cmath` for Ruby 2.7 + and later. +- Silence the warnings about an uninitialized `$GSL`, and correct the rb-gsl + URL hint. +- Package the test files in the gem. +- Add a Gemfile.lock. + +## 1.3.5 - 2018-04-17 + +- Use Minitest for the test suite. +- Add the `mathn` dependency, which Ruby 2.5.0 removed. +- Make the gem installable through Bundler and a git remote. +- Fix the gemspec and the unit tests. + +## 1.3.4 - 2013-12-31 + +- Use a prior in the Bayes classifier. +- Change the skip word list from an array to a set. +- Reduce the number of regular expression matches during tokenization. +- Stem only the words that the tokenizer keeps. +- Default the word hash values to 0. +- Add a gemspec, a README, and Travis CI. + +## 1.3.3 - 2010-07-06 + +- Use fast-stemmer for the Porter stemmer. +- Check `$GSL` before a call to `Matrix.diag`, so the code uses `GSL::Matrix`. + +## 1.3.2 - 2010-07-06 + +- Fix the reported issue #1. + +## 1.3.1 and earlier + +Versions 1.0 through 1.3.1 reached RubyGems on 2009-07-25. The git history +before 2010 is too sparse to attribute each change to one of these versions. diff --git a/CLAUDE.md b/CLAUDE.md index efec2996..43c994c2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,77 +1 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## Project Overview - -Ruby gem providing text classification via two algorithms: -- **Bayes** (`Classifier::Bayes`) - Naive Bayesian classification -- **LSI** (`Classifier::LSI`) - Latent Semantic Indexing for semantic classification, clustering, and search - -## Common Commands - -```bash -# Compile native C extension -bundle exec rake compile - -# Run all tests (compiles first) -bundle exec rake test - -# Run a single test file -ruby -Ilib test/bayes/bayesian_test.rb -ruby -Ilib test/lsi/lsi_test.rb - -# Run tests with pure Ruby (no native extension) -NATIVE_VECTOR=true bundle exec rake test - -# Run benchmarks -bundle exec rake benchmark -bundle exec rake benchmark:compare - -# Interactive console -bundle exec rake console - -# Generate documentation -bundle exec rake doc -``` - -## Architecture - -### Core Components - -**Bayesian Classifier** (`lib/classifier/bayes.rb`) -- Train with `train(category, text)` or dynamic methods like `train_spam(text)` -- Classify with `classify(text)` returning the best category -- Uses log probabilities for numerical stability - -**LSI Classifier** (`lib/classifier/lsi.rb`) -- Uses Singular Value Decomposition (SVD) for semantic analysis -- Native C extension for 5-50x faster matrix operations; falls back to pure Ruby -- Key operations: `add_item`, `classify`, `find_related`, `search` -- `auto_rebuild` option controls automatic index rebuilding after changes - -**String Extensions** (`lib/classifier/extensions/word_hash.rb`) -- `word_hash` / `clean_word_hash` - tokenize text to stemmed word frequencies -- `CORPUS_SKIP_WORDS` - stopwords filtered during tokenization -- Uses `fast-stemmer` gem for Porter stemming - -**Vector Extensions** (`lib/classifier/extensions/vector.rb`) -- Pure Ruby SVD implementation (`Matrix#SV_decomp`) - used as fallback -- Vector normalization and magnitude calculations - -### Native C Extension (`ext/classifier/`) - -LSI uses a native C extension for fast linear algebra operations: -- `Classifier::Linalg::Vector` - Vector operations (alloc, normalize, dot product) -- `Classifier::Linalg::Matrix` - Matrix operations (alloc, transpose, multiply) -- Jacobi SVD implementation for singular value decomposition - -Check current backend: `Classifier::LSI.backend` returns `:native` or `:ruby` -Force pure Ruby: `NATIVE_VECTOR=true bundle exec rake test` - -### Content Nodes (`lib/classifier/lsi/content_node.rb`) - -Internal data structure storing: -- `word_hash` - term frequencies -- `raw_vector` / `raw_norm` - initial vector representation -- `lsi_vector` / `lsi_norm` - reduced dimensionality representation after SVD +@AGENTS.md diff --git a/Gemfile.lock b/Gemfile.lock index fb0a83c9..2932c42c 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,7 +1,7 @@ PATH remote: . specs: - classifier (2.6.0) + classifier (2.7.0) fast-stemmer (~> 1.0) matrix mutex_m (~> 0.2) diff --git a/classifier.gemspec b/classifier.gemspec index 0a0a4321..a8b6ee0d 100644 --- a/classifier.gemspec +++ b/classifier.gemspec @@ -15,10 +15,11 @@ Gem::Specification.new do |s| 'documentation_uri' => 'https://rubyclassifier.com/docs', 'source_code_uri' => 'https://github.com/cardmagic/classifier', 'bug_tracker_uri' => 'https://github.com/cardmagic/classifier/issues', - 'changelog_uri' => 'https://github.com/cardmagic/classifier/releases' + 'changelog_uri' => 'https://github.com/cardmagic/classifier/blob/master/CHANGELOG.md' } s.required_ruby_version = '>= 3.1' - s.files = Dir['{lib,sig,exe}/**/*.{rb,rbs}', 'ext/**/*.{c,h,rb}', 'exe/*', 'bin/*', 'LICENSE', '*.md', 'test/*'] + s.files = Dir['{lib,sig,exe}/**/*.{rb,rbs}', 'ext/**/*.{c,h,rb}', 'exe/*', 'bin/*', 'LICENSE', + 'README.md', 'CHANGELOG.md', 'test/*'] s.bindir = 'exe' s.executables = %w[classifier keywords] s.extensions = ['ext/classifier/extconf.rb'] diff --git a/lib/classifier/version.rb b/lib/classifier/version.rb index 861b19ff..94994168 100644 --- a/lib/classifier/version.rb +++ b/lib/classifier/version.rb @@ -1,3 +1,3 @@ module Classifier - VERSION = '2.6.0'.freeze + VERSION = '2.7.0'.freeze end