From 3752aac6b1eca1517716fa3e7fed3b9c5436c70a Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Sat, 15 Aug 2026 10:31:23 -0700 Subject: [PATCH 1/2] docs: show the two commands working together Neither README.md nor docs/cli.md said how classifier and keywords relate. Each was documented alone, so a reader met two tools and no reason to run them together. Add three workflows, each verified against the installed 2.7.0 binaries: Read the label and the reason side by side. classifier -p gives the verdict and keywords -n gives the terms that carried it, which is what you want when a classification surprises you. This is the pattern the explainability literature calls showing top terms alongside the predicted label. Find corpus-specific stopwords. A review corpus repeats "delivery" and no general stopword list knows it. --max-df drops a term above a document ratio, so comparing keywords info before and after names the term, and dropping it sharpens every term that carries signal. Keep the models straight. The two commands write different formats and reject each other, classifier takes -f, and keywords takes -m. Also warn against piping keywords into classifier. TF-IDF ranks a term by how well it separates one document from the corpus, not by how well it signals a category, so the top terms are not the strongest evidence. Measured on a long review, the full text classifies at 0.92 and its top four terms at 0.79. --- README.md | 15 +++++- docs/cli.md | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c2a93f5..6f8d0eb 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,20 @@ space, as in `machine learning:0.35`. Run `keywords --help` for the full option list. A usage error exits 2 and any other error exits 1, so scripts can tell the two apart. -[keywords reference →](docs/keywords.md) · [CLI Guide →](https://rubyclassifier.com/docs/guides/cli/basics) +Run the two commands side by side to get the label and the reason for it: + +```bash +classifier -f reviews-model.json -p "Broken on arrival, awful quality" +# => positive:0.12 negative:0.88 + +keywords -m reviews.json -n 5 "Broken on arrival, awful quality" +# => awful:0.52 arrival:0.52 broken:0.52 quality:0.44 +``` + +They keep separate models in separate formats, so `classifier` takes `-f` and +`keywords` takes `-m`, and neither reads the other's file. + +[Using both commands →](docs/cli.md#using-both-commands-together) · [keywords reference →](docs/keywords.md) · [CLI Guide →](https://rubyclassifier.com/docs/guides/cli/basics) ### Claude Code Plugin diff --git a/docs/cli.md b/docs/cli.md index afda9f5..83487f7 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -135,6 +135,137 @@ $ classifier -m lsi related article.txt | `-v`, `--version` | Print the gem version | | `-h`, `--help` | Print the full usage | +## Using both commands together + +`classifier` and `keywords` answer different questions about the same text. +`classifier` gives a category. `keywords` gives the terms that carry the text. +Run them side by side to see the label and the reason behind it. + +Fit a vocabulary from the same corpus you train on, and the two views line up: + +```console +$ keywords fit -m reviews.json reviews/good/*.txt reviews/bad/*.txt +$ keywords info -m reviews.json +Documents: 8 +Vocabulary: 37 +Min DF: 1 +Max DF: 1.0 +``` + +`keywords info` reports the corpus before you commit to training. A vocabulary +of 37 terms over 8 documents says the corpus is far too small, and no +classifier fixes that. + +Then train, and read the two answers together: + +```console +$ classifier -f reviews-model.json train positive reviews/good/*.txt +$ classifier -f reviews-model.json train negative reviews/bad/*.txt + +$ classifier -f reviews-model.json -p "Broken on arrival, awful quality and useless customer service" +positive:0.07 negative:0.93 + +$ keywords -m reviews.json -n 5 "Broken on arrival, awful quality and useless customer service" +useless:0.4 awful:0.4 arrival:0.4 broken:0.4 service:0.34 +``` + +The first line is the verdict. The second says which terms drove it, which is +what you need when a classification surprises you. + +### Find the words your corpus wastes on itself + +A term that appears in nearly every document tells a classifier nothing, and +every corpus grows its own. A review corpus repeats `delivery`, a support +corpus repeats `ticket`. These are stopwords that no general stopword list +knows about, because they are specific to your data. + +`keywords` finds them, because `--max-df` drops a term that appears in more +than the given ratio of documents. Compare the vocabulary size before and +after: + +```console +$ keywords fit -m default.json good.txt bad.txt +$ keywords info -m default.json +Documents: 12 +Vocabulary: 42 + +$ keywords fit -m pruned.json --max-df 0.5 good.txt bad.txt +$ keywords info -m pruned.json +Documents: 12 +Vocabulary: 41 +``` + +One term went. Score a document under each model to see which: + +```console +$ keywords -m default.json -n 3 "delivery was awful and broken" +broken:0.69 awful:0.69 delivery:0.24 + +$ keywords -m pruned.json -n 3 "delivery was awful and broken" +broken:0.71 awful:0.71 +``` + +`delivery` sat in all 12 documents and still drew weight. Dropping it sharpens +every term that carries real signal. + +Watch the vocabulary count as you tune, because these bounds cut fast: + +```console +$ keywords fit -m tight.json --min-df 2 good.txt bad.txt +$ keywords info -m tight.json +Documents: 12 +Vocabulary: 7 +``` + +`--min-df 2` took 42 terms down to 7. That is no longer a vocabulary, it is a +handful of words. Move one bound at a time and read `keywords info` after each +change. + +### Two commands, two models + +The models are separate files in separate formats, and neither command reads +the other's: + +```console +$ classifier -f reviews.json "broken awful" +Error: Unknown classifier type in model: tfidf + +$ keywords -m reviews-model.json "broken awful" +Error: Invalid vectorizer type: bayes +``` + +Note the flags differ too. `classifier` takes `-f`, and `keywords` takes `-m`. + +| | `classifier` | `keywords` | +|:--|:--|:--| +| Answers | Which category | Which terms matter | +| Model flag | `-f` | `-m` | +| Default model | `./classifier.json` | `./keywords.json` | +| Builds a model with | `train` | `fit` | +| Pre-trained models | Yes, through `-r` | No | + +### Do not pipe one into the other + +`keywords` prints `term:score` pairs, which is not text to classify. Feeding +its output to `classifier` throws away the rest of the document and weakens the +result: + +```console +$ classifier -f reviews-model.json -p "$LONG_REVIEW" +positive:0.08 negative:0.92 # the whole review + +$ keywords -m reviews.json -n 4 "$LONG_REVIEW" +arrived:0.6 refund:0.3 useless:0.3 build:0.3 + +$ classifier -f reviews-model.json -p "arrived refund useless build" +positive:0.21 negative:0.79 # weaker, from the top terms alone +``` + +Confidence drops from 0.92 to 0.79. TF-IDF ranks a term by how much it +distinguishes one document from the rest of the corpus, which is not the same +as how much it signals a category. Here it puts `arrived` first, a neutral +word about delivery. Classify the full text and use `keywords` to explain it. + ## Install without Ruby Homebrew installs the command line tools on their own: From a585814c866a44a3fd675e8544152096e063b88d Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Sat, 15 Aug 2026 10:46:08 -0700 Subject: [PATCH 2/2] docs: stop calling TF-IDF terms an explanation The side-by-side section said the keywords output showed "which terms drove" the classifier verdict. It does not. The two commands hold separate models. keywords reports TF-IDF weight, which measures how well a term separates one document from its corpus, and it never sees the classifier or learns which category a term favors. A term can top that list and carry no weight in the decision. Presenting it as attribution invites the reader to tune the wrong thing. Say what the terms are: the document in shorthand, useful for checking that it says what you assumed. Point at Classifier::LogisticRegression#weights for the weights a model actually holds, and note that no command line flag reports per-term weights for a Bayes model. Found by Greptile on #172. --- README.md | 7 +++++-- docs/cli.md | 30 +++++++++++++++++++++++++----- 2 files changed, 30 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 6f8d0eb..3cc2365 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,8 @@ space, as in `machine learning:0.35`. Run `keywords --help` for the full option list. A usage error exits 2 and any other error exits 1, so scripts can tell the two apart. -Run the two commands side by side to get the label and the reason for it: +Run the two commands side by side to read a label together with the terms that +make the text distinctive: ```bash classifier -f reviews-model.json -p "Broken on arrival, awful quality" @@ -129,7 +130,9 @@ keywords -m reviews.json -n 5 "Broken on arrival, awful quality" ``` They keep separate models in separate formats, so `classifier` takes `-f` and -`keywords` takes `-m`, and neither reads the other's file. +`keywords` takes `-m`, and neither reads the other's file. The terms are +context, not an explanation of the label: TF-IDF measures how well a term +separates a document from its corpus, not how much it favors a category. [Using both commands →](docs/cli.md#using-both-commands-together) · [keywords reference →](docs/keywords.md) · [CLI Guide →](https://rubyclassifier.com/docs/guides/cli/basics) diff --git a/docs/cli.md b/docs/cli.md index 83487f7..2c38abe 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -138,8 +138,9 @@ $ classifier -m lsi related article.txt ## Using both commands together `classifier` and `keywords` answer different questions about the same text. -`classifier` gives a category. `keywords` gives the terms that carry the text. -Run them side by side to see the label and the reason behind it. +`classifier` gives a category. `keywords` names the terms that make the text +distinctive against its corpus. Run them side by side to read a label together +with what the document is about. Fit a vocabulary from the same corpus you train on, and the two views line up: @@ -169,8 +170,26 @@ $ keywords -m reviews.json -n 5 "Broken on arrival, awful quality and useless cu useless:0.4 awful:0.4 arrival:0.4 broken:0.4 service:0.34 ``` -The first line is the verdict. The second says which terms drove it, which is -what you need when a classification surprises you. +The first line is the verdict. The second is the document in shorthand, which +tells you what the classifier was reading when a result surprises you. + +**The second line is not an explanation of the first.** The two commands hold +separate models. `keywords` reports TF-IDF weight, which measures how well a +term separates this document from the rest of the corpus. It never sees the +classifier, and it does not know which category a term favors. A term can top +the list and carry no weight in the decision. + +Read it as context, not as attribution. When a label looks wrong, the terms +tell you whether the document says what you assumed, which is usually the real +problem. For the weights a model actually holds, use +`Classifier::LogisticRegression#weights` from Ruby, which returns the learned +weight per term and per category: + +```ruby +classifier.weights("positive", limit: 5) +``` + +No command line flag reports per-term weights for a Bayes model. ### Find the words your corpus wastes on itself @@ -264,7 +283,8 @@ positive:0.21 negative:0.79 # weaker, from the top terms alone Confidence drops from 0.92 to 0.79. TF-IDF ranks a term by how much it distinguishes one document from the rest of the corpus, which is not the same as how much it signals a category. Here it puts `arrived` first, a neutral -word about delivery. Classify the full text and use `keywords` to explain it. +word about delivery. Classify the full text, and read `keywords` alongside it +for context rather than for attribution. ## Install without Ruby