From ce36103e5fa9fa19e4430f7b1ca8a4523fd7d04b Mon Sep 17 00:00:00 2001 From: Bethany Moore Date: Tue, 8 Sep 2026 11:26:31 -0400 Subject: [PATCH 1/9] Create understanding_draft_quality.md --- understanding_draft_quality.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 understanding_draft_quality.md diff --git a/understanding_draft_quality.md b/understanding_draft_quality.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/understanding_draft_quality.md @@ -0,0 +1 @@ + From f08390fd84065d2f49b9954691a76379b1dce5c7 Mon Sep 17 00:00:00 2001 From: Bethany Moore Date: Tue, 8 Sep 2026 11:29:27 -0400 Subject: [PATCH 2/9] Create understanding_draft_quality.md Create in correct location --- docs/Draft-Generation/understanding_draft_quality.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/Draft-Generation/understanding_draft_quality.md diff --git a/docs/Draft-Generation/understanding_draft_quality.md b/docs/Draft-Generation/understanding_draft_quality.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/docs/Draft-Generation/understanding_draft_quality.md @@ -0,0 +1 @@ + From d94b1d0c3ab067ab187c359e3bd615b0d3b867a7 Mon Sep 17 00:00:00 2001 From: Bethany Moore Date: Tue, 8 Sep 2026 15:42:37 -0400 Subject: [PATCH 3/9] Update understanding_draft_quality.md add understanding draft quality text --- .../understanding_draft_quality.md | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/Draft-Generation/understanding_draft_quality.md b/docs/Draft-Generation/understanding_draft_quality.md index 8b13789..512657d 100644 --- a/docs/Draft-Generation/understanding_draft_quality.md +++ b/docs/Draft-Generation/understanding_draft_quality.md @@ -1 +1,40 @@ +--- +title: Understanding Draft Quality +sidebar_position: 1 +slug: /understanding-draft-quality +--- +## Elements of Draft Quality {} + +Only human translators can decide whether a draft serves a team’s needs. However, it is helpful to understand some factors that can influence the quality of a generated draft. + +- Amount and quality of training data (language examples) + - More examples contribute to better draft quality. + - Examples that are well-checked and consistent improve draft quality. + - Examples that are relevant to the book or books to be drafted improve draft quality; for example, including examples in the same genre or with similar content is often helpful. +- Relationship between source and target languages + - A closer linguistic relationship and typological match often results in better draft quality. +- Relationship between source text and target texts + - When the translation closely follows the words of the reference text, this can make model training more successful and improve draft quality. + - For this reason, including a back translation in the source language as a second reference text may improve quality. +- Representation of source language in the language model + - Using a high-resource source language, that is, one which is a major modern language and has plentiful examples in the underlying language model, can improve draft quality. +- Characteristics of target project + - A script unknown to the underlying language model may decrease draft quality. + - Agglutinative languages or those without word breaks may experience decreased draft quality. + - A project that translates at the paragraph level, uses many long verse ranges, or includes significant explication not found in the source text may see decreased draft quality. + +## Improving Draft Quality {} + +Here are some ideas to improve draft quality. Not all of these will be possible or relevant for all projects. +- Use as much translated Scripture text as possible for training the model. +- Consider adding parallel linguistic data in the source and target languages. +- Test different reference texts, including higher-resource or more closely-related source languages. +- Include a back translation in the source language as a second reference text. +- Correct any frequent spelling, keyboarding, or vocabulary inconsistencies in the translation. +- Continue with manual translation and test draft generation again when more translated Scripture is complete. +- Use [Incremental Draft Generation]() when beginning a new genre, such as with Psalms. + +## Draft Quality Warning {} + +Scripture Forge uses automated scoring to estimate draft quality, and will present a warning to users if the estimated quality of a generated draft is very low. This may indicate that there is an error in the configuration that resulted in unsuccessful model training. Please see the FAQ for more information. Some drafts may be low quality even if they are not flagged by the warning, so it is important to check the quality of each draft carefully. From fc0f48ea1d021dc2e06e58138b68d6e513565be1 Mon Sep 17 00:00:00 2001 From: Bethany Moore Date: Tue, 8 Sep 2026 16:15:57 -0400 Subject: [PATCH 4/9] Update understanding_draft_quality.md Add GUIDs for page sections --- docs/Draft-Generation/understanding_draft_quality.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Draft-Generation/understanding_draft_quality.md b/docs/Draft-Generation/understanding_draft_quality.md index 512657d..af195e6 100644 --- a/docs/Draft-Generation/understanding_draft_quality.md +++ b/docs/Draft-Generation/understanding_draft_quality.md @@ -4,7 +4,7 @@ sidebar_position: 1 slug: /understanding-draft-quality --- -## Elements of Draft Quality {} +## Elements of Draft Quality Only human translators can decide whether a draft serves a team’s needs. However, it is helpful to understand some factors that can influence the quality of a generated draft. @@ -24,7 +24,7 @@ Only human translators can decide whether a draft serves a team’s needs. Howev - Agglutinative languages or those without word breaks may experience decreased draft quality. - A project that translates at the paragraph level, uses many long verse ranges, or includes significant explication not found in the source text may see decreased draft quality. -## Improving Draft Quality {} +## Improving Draft Quality {#92f8d5c800084e4eb113511f74e26927} Here are some ideas to improve draft quality. Not all of these will be possible or relevant for all projects. - Use as much translated Scripture text as possible for training the model. @@ -35,6 +35,6 @@ Here are some ideas to improve draft quality. Not all of these will be possible - Continue with manual translation and test draft generation again when more translated Scripture is complete. - Use [Incremental Draft Generation]() when beginning a new genre, such as with Psalms. -## Draft Quality Warning {} +## Draft Quality Warning {#9a12a839f3634e75acd511f4b426eb86} Scripture Forge uses automated scoring to estimate draft quality, and will present a warning to users if the estimated quality of a generated draft is very low. This may indicate that there is an error in the configuration that resulted in unsuccessful model training. Please see the FAQ for more information. Some drafts may be low quality even if they are not flagged by the warning, so it is important to check the quality of each draft carefully. From bc4746a247704c6f4d5cf65546e34b5305654758 Mon Sep 17 00:00:00 2001 From: Bethany Moore Date: Tue, 8 Sep 2026 16:27:23 -0400 Subject: [PATCH 5/9] Add FAQs for quality warning.md --- docs/faq.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/faq.md b/docs/faq.md index 26c28a8..b40b336 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -76,3 +76,35 @@ slug: /faq 6. Why is my book introduction cut short? _Scripture Forge uses the NLLB model to generate drafts. NLLB is a sentence model, and it is most accurate with short segments of text. To maximize accuracy, there is a limit of 200 tokens (word parts) per translated segment, and any text after that point is cut off. Scripture verses are translated one-by-one and very rarely exceed this limit. Paragraph-length book introductions that are longer than the allowed length will be cut off short, and only the first portion will be translated._ + +## Questions on draft quality warnings {#f47ac10b58cc4372a5670e02b2c3d479} + +1. Why do I see a warning that my draft may have lower than normal quality? What should I do if I see this warning? + + _Scripture Forge uses automated metrics to determine whether model training was successful enough to prepare a draft of the requested books. Sometimes these metrics indicate that draft quality may be poor._ + + _The most common reason for this warning is that there is not enough example text of the kind needed to train the model. Usually, this situation results from an error in setting up the draft request. Use the following checklist as a guide to check your draft generation configuration._ + - _Project: Are you generating the draft from a project that includes all completed books in the project?_ + - _It is important not to request drafts from a project used to store AI drafts._ + - _If some translated books, such as the New Testament, are stored in a separate Paratext project, you may need to create a new project and import all translated books to that project. You can use that project for draft generation._ + - _Language codes: Does the language code for each reference and source project match the text of that project?_ + - _You can update language codes in Paratext under “Project properties.”_ + - _Draft Books: Are the book or books to draft selected in Step 2, “Select books to draft”?_ + - _Training Books: Are all the completed and checked books in the project selected in Step 3, “Select books to train on”?_ + - _Be sure all the books used for training have been checked by the team. They should be generally consistent in spelling, character use (keyboarding), dialect, and vocabulary._ + - _Be sure the reference text contains all the books used for training, and that they are selected at the bottom of Step 3._ + - _Back Translations: Are you using a back translation as a reference project? If so, check these things:_ + - _Be sure the back translation language code represents the language of the back translation and not of the main translation project._ + - _If any books in the back translation project are in a different language than the rest of the back translation, unselect those books from the back translation area at the bottom of Step 3._ + - _Additional Data: Is it formatted correctly?_ + - _If you uploaded additional data for model training on the “Configure sources” page, be sure that each row of that file contains sentences with the same meaning, with the source language on the left and the target language on the right._ + +2. Why do I still see a low draft quality warning after completing all the checks? + + _Remember that draft quality is only an estimate and may not reflect human judgment. Please review the draft carefully to determine whether it is useful in your translation work. The help text about [Understanding Draft Quality](https://help.scriptureforge.org/understanding-draft-quality) may be helpful._ + + _If draft quality is still low after completing the recommended checks, you may need to translate more text before generating a draft. Early in a translation project, the choice of which book to draft next is important. For example, a model trained on the book of Luke is better prepared to draft Mark than to draft Romans. If you are working in the New Testament, please review the [Guidelines for New Testament Draft Generation](https://help.scriptureforge.org/new-testament-drafting) to learn about how book sequence can affect draft quality._ + +3. What should I do if only some of the books I generated have a low draft quality warning? + + _A warning on any draft is a good reason to consider the checklist above. However, it may also be the case that your training data was a better fit for drafting some books than others. It is important to review each book draft carefully, especially those with a low quality warning, before deciding whether to use it in your translation process._ From 753a54c7049f2018850fe926a25fa0b3592e923e Mon Sep 17 00:00:00 2001 From: Bethany Moore Date: Tue, 8 Sep 2026 17:35:21 -0400 Subject: [PATCH 6/9] Insert correct link to incremental draft generation --- docs/Draft-Generation/understanding_draft_quality.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Draft-Generation/understanding_draft_quality.md b/docs/Draft-Generation/understanding_draft_quality.md index af195e6..30e9bea 100644 --- a/docs/Draft-Generation/understanding_draft_quality.md +++ b/docs/Draft-Generation/understanding_draft_quality.md @@ -33,8 +33,8 @@ Here are some ideas to improve draft quality. Not all of these will be possible - Include a back translation in the source language as a second reference text. - Correct any frequent spelling, keyboarding, or vocabulary inconsistencies in the translation. - Continue with manual translation and test draft generation again when more translated Scripture is complete. -- Use [Incremental Draft Generation]() when beginning a new genre, such as with Psalms. +- See the note about [Incremental Draft Generation](https://help.scriptureforge.org/generating-a-draft#select-the-books-to-draft) and consider using this method for long books or when beginning a new genre. ## Draft Quality Warning {#9a12a839f3634e75acd511f4b426eb86} -Scripture Forge uses automated scoring to estimate draft quality, and will present a warning to users if the estimated quality of a generated draft is very low. This may indicate that there is an error in the configuration that resulted in unsuccessful model training. Please see the FAQ for more information. Some drafts may be low quality even if they are not flagged by the warning, so it is important to check the quality of each draft carefully. +Scripture Forge uses automated scoring to estimate draft quality, and will present a warning to users if the estimated quality of a generated draft is very low. This may indicate that there is an error in the configuration that resulted in unsuccessful model training. Please see the FAQ for more information. Some drafts may be low quality even if they are not flagged by the warning. It is important to check the quality of each draft carefully. From 9664639e37307b4c3b6f2258a3ca34003e191bcc Mon Sep 17 00:00:00 2001 From: Bethany Moore Date: Tue, 22 Sep 2026 16:54:45 -0400 Subject: [PATCH 7/9] Delete extra file understanding_draft_quality.md --- understanding_draft_quality.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 understanding_draft_quality.md diff --git a/understanding_draft_quality.md b/understanding_draft_quality.md deleted file mode 100644 index 8b13789..0000000 --- a/understanding_draft_quality.md +++ /dev/null @@ -1 +0,0 @@ - From 7daa941a79f01698f31452f4627c3e8c314df8e6 Mon Sep 17 00:00:00 2001 From: Bethany Moore Date: Tue, 22 Sep 2026 17:10:09 -0400 Subject: [PATCH 8/9] Address comments on understanding_draft_quality.md --- docs/Draft-Generation/understanding_draft_quality.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Draft-Generation/understanding_draft_quality.md b/docs/Draft-Generation/understanding_draft_quality.md index 30e9bea..9aec890 100644 --- a/docs/Draft-Generation/understanding_draft_quality.md +++ b/docs/Draft-Generation/understanding_draft_quality.md @@ -1,10 +1,10 @@ --- -title: Understanding Draft Quality +title: Improving Draft Quality sidebar_position: 1 -slug: /understanding-draft-quality +slug: /improving-draft-quality --- -## Elements of Draft Quality +## Factors that Affect Draft Quality Only human translators can decide whether a draft serves a team’s needs. However, it is helpful to understand some factors that can influence the quality of a generated draft. @@ -28,7 +28,7 @@ Only human translators can decide whether a draft serves a team’s needs. Howev Here are some ideas to improve draft quality. Not all of these will be possible or relevant for all projects. - Use as much translated Scripture text as possible for training the model. -- Consider adding parallel linguistic data in the source and target languages. +- Consider uploading a file with additional paired sentences in the source and target languages. This can be done from the "Configure sources" page. - Test different reference texts, including higher-resource or more closely-related source languages. - Include a back translation in the source language as a second reference text. - Correct any frequent spelling, keyboarding, or vocabulary inconsistencies in the translation. From e48f2be4047422f5b10b24dee40b507a6378b5f1 Mon Sep 17 00:00:00 2001 From: Bethany Moore Date: Tue, 22 Sep 2026 17:10:57 -0400 Subject: [PATCH 9/9] Rename understanding_draft_quality.md to improving_draft_quality.md --- ...{understanding_draft_quality.md => improving_draft_quality.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/Draft-Generation/{understanding_draft_quality.md => improving_draft_quality.md} (100%) diff --git a/docs/Draft-Generation/understanding_draft_quality.md b/docs/Draft-Generation/improving_draft_quality.md similarity index 100% rename from docs/Draft-Generation/understanding_draft_quality.md rename to docs/Draft-Generation/improving_draft_quality.md