Draft quality warning by bhartmoore · Pull Request #64 · sillsdev/scripture-forge-help

bhartmoore · 2026-09-08T20:29:58Z

Create help text and add to FAQ section to explain draft quality warnings.

Create in correct location

add understanding draft quality text

Add GUIDs for page sections

github-actions · 2026-09-08T21:38:56Z

🚀 Preview deployed!

You can preview this PR at: https://pr-64--sf-help-preview.netlify.app

github-actions · 2026-09-08T21:38:56Z

🚀 Preview deployed!

You can preview this PR at: https://pr-64--sf-help-preview.netlify.app

Nateowami · 2026-09-10T18:33:04Z

+
+Only human translators can decide whether a draft serves a team’s needs. However, it is helpful to understand some factors that can influence the quality of a generated draft.
+
+- Amount and quality of training data (language examples)


I think it would be helpful to emphasize we're talking about paired data. If a user is reading this and doesn't already understand that concept that more training data is generally better, they almost certainly don't understand the need for training data to be paired.

Nateowami · 2026-09-10T18:33:07Z

+  - A closer linguistic relationship and typological match often results in better draft quality. 
+- Relationship between source text and target texts
+  - When the translation closely follows the words of the reference text, this can make model training more successful and improve draft quality. 
+  - For this reason, including a back translation in the source language as a second reference text may improve quality. 


"For this reason" makes it feel like it's a continuation of the point above. Maybe that's OK and better than one really long point?

It seems like using a back translation as a reference may be a confusing concept and might warrant an FAQ entry that we link to.

Nateowami · 2026-09-10T18:33:09Z

+  - When the translation closely follows the words of the reference text, this can make model training more successful and improve draft quality. 
+  - For this reason, including a back translation in the source language as a second reference text may improve quality. 
+- Representation of source language in the language model
+  - Using a high-resource source language, that is, one which is a major modern language and has plentiful examples in the underlying language model, can improve draft quality.


Maybe parentheses instead of "that is"? It seems like it would make it easier to parse the sentence.

Nateowami · 2026-09-10T18:35:42Z

+slug: /understanding-draft-quality
+---
+
+## Elements of Draft Quality


The title says "Elements of Draft Quality", but I think it's really more so "factors that impact quality".

I think "elements of" would be things like word choice, naturalness, accuracy, spelling, punctuation, etc.

Nateowami · 2026-09-10T18:37:29Z

+
+Here are some ideas to improve draft quality. Not all of these will be possible or relevant for all projects.
+- Use as much translated Scripture text as possible for training the model. 
+- Consider adding parallel linguistic data in the source and target languages.


Maybe word this in a way that suggests how, such as "including files with paired sentences" rather than "parallel linguistic data"? It feels like less jargon, and also implies how it's done.

Nateowami · 2026-09-10T18:38:57Z

+
+## Draft Quality Warning {#9a12a839f3634e75acd511f4b426eb86}
+
+Scripture Forge uses automated scoring to estimate draft quality, and will present a warning to users if the estimated quality of a generated draft is very low. This may indicate that there is an error in the configuration that resulted in unsuccessful model training. Please see the FAQ for more information. Some drafts may be low quality even if they are not flagged by the warning. It is important to check the quality of each draft carefully.


We'll need to make sure we don't publish this until this feature goes live. Probably won't be a problem as that's likely next week. Just leaving this as a note so I don't forget.

Nateowami · 2026-09-10T18:40:33Z

@@ -0,0 +1,40 @@
+---
+title: Understanding Draft Quality


I wonder if "Improving Draft Quality" might be a better title?

I know the "understanding" wording mirrors the "Understanding Draft Generation" title I wrote, but I can't really think of any alternative way to title that document; it's very much a "how it works" document, rather than a "things to try" document.

Nateowami · 2026-09-10T18:41:45Z

@@ -0,0 +1,40 @@
+---
+title: Understanding Draft Quality
+sidebar_position: 1


Is this where you are intending to make the document land in the sidebar?

Nateowami · 2026-09-10T18:53:29Z

+  - A script unknown to the underlying language model may decrease draft quality.
+  - Agglutinative languages or those without word breaks may experience decreased draft quality.
+  - A project that translates at the paragraph level, uses many long verse ranges, or includes significant explication not found in the source text may see decreased draft quality.
+


I wonder if this might be more readable if the sections were minor headings instead of bullet points? What do you think?

What it currently looks like

Alternate example A

Alternate example B

Nateowami · 2026-09-10T18:57:39Z

+
+   _Remember that draft quality is only an estimate and may not reflect human judgment. Please review the draft carefully to determine whether it is useful in your translation work. The help text about [Understanding Draft Quality](https://help.scriptureforge.org/understanding-draft-quality) may be helpful._
+
+   _If draft quality is still low after completing the recommended checks, you may need to translate more text before generating a draft. Early in a translation project, the choice of which book to draft next is important. For example, a model trained on the book of Luke is better prepared to draft Mark than to draft Romans. If you are working in the New Testament, please review the [Guidelines for New Testament Draft Generation](https://help.scriptureforge.org/new-testament-drafting) to learn about how book sequence can affect draft quality._


Maybe also suggest reaching out for help?

Nateowami

Sorry for so many comments.

Nateowami · 2026-09-10T18:57:45Z

+
+2. Why do I still see a low draft quality warning after completing all the checks?
+
+   _Remember that draft quality is only an estimate and may not reflect human judgment. Please review the draft carefully to determine whether it is useful in your translation work. The help text about [Understanding Draft Quality](https://help.scriptureforge.org/understanding-draft-quality) may be helpful._


draft quality is only an estimate

I think it's supposed to say something like "the warning is based on an estimate".

Nateowami · 2026-09-10T18:58:27Z

+   - _Language codes: Does the language code for each reference and source project match the text of that project?_ 
+      - _You can update language codes in Paratext under “Project properties.”_
+   - _Draft Books: Are the book or books to draft selected in Step 2, “Select books to draft”?_ 
+   - _Training Books: Are all the completed and checked books in the project selected in Step 3, “Select books to train on”?_ 


This seems to be missing any note about making sure the reference text training data has content and is selected. We auto select it when you select target books, but it's possible for it to be essentially blank.

Nateowami · 2026-09-10T18:58:52Z

@@ -0,0 +1 @@
+


I think this file was created by accident.

bhartmoore added 5 commits September 8, 2026 11:26

Create understanding_draft_quality.md

125a0c4

Create understanding_draft_quality.md

f4ef65c

Create in correct location

Update understanding_draft_quality.md

7cf4413

add understanding draft quality text

Update understanding_draft_quality.md

4c5c561

Add GUIDs for page sections

Add FAQs for quality warning.md

10e5334

bhartmoore had a problem deploying to preview September 8, 2026 20:30 — with GitHub Actions Failure

bhartmoore closed this Sep 8, 2026

Insert correct link to incremental draft generation

72b50d0

bhartmoore reopened this Sep 8, 2026

bhartmoore deployed to preview September 8, 2026 21:35 — with GitHub Actions Active

Nateowami reviewed Sep 10, 2026

View reviewed changes

Nateowami requested changes Sep 10, 2026

View reviewed changes

pmachapman mentioned this pull request Sep 14, 2026

SF-3917 Add a link to Low Draft Quality banner sillsdev/web-xforge#4104

Open


		Only human translators can decide whether a draft serves a team’s needs. However, it is helpful to understand some factors that can influence the quality of a generated draft.

		- Amount and quality of training data (language examples)


		## Draft Quality Warning {#9a12a839f3634e75acd511f4b426eb86}

		Scripture Forge uses automated scoring to estimate draft quality, and will present a warning to users if the estimated quality of a generated draft is very low. This may indicate that there is an error in the configuration that resulted in unsuccessful model training. Please see the FAQ for more information. Some drafts may be low quality even if they are not flagged by the warning. It is important to check the quality of each draft carefully.


		_Remember that draft quality is only an estimate and may not reflect human judgment. Please review the draft carefully to determine whether it is useful in your translation work. The help text about [Understanding Draft Quality](https://help.scriptureforge.org/understanding-draft-quality) may be helpful._

		_If draft quality is still low after completing the recommended checks, you may need to translate more text before generating a draft. Early in a translation project, the choice of which book to draft next is important. For example, a model trained on the book of Luke is better prepared to draft Mark than to draft Romans. If you are working in the New Testament, please review the [Guidelines for New Testament Draft Generation](https://help.scriptureforge.org/new-testament-drafting) to learn about how book sequence can affect draft quality._


		2. Why do I still see a low draft quality warning after completing all the checks?

		_Remember that draft quality is only an estimate and may not reflect human judgment. Please review the draft carefully to determine whether it is useful in your translation work. The help text about [Understanding Draft Quality](https://help.scriptureforge.org/understanding-draft-quality) may be helpful._

Uh oh!

Conversation

bhartmoore commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nateowami Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

What it currently looks like

Alternate example A

Alternate example B

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nateowami left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Nateowami Sep 10, 2026 •

edited

Loading