ASH: A Systematic Automated Evaluation Framework for Creative Text Generation in Cuisine Transfer Tasks
Accepted for publication in IEEE Access
This repository provides the code and data for "ASH: A Systematic Automated Evaluation Framework for Creative Text Generation in Cuisine Transfer Tasks". This research extends our previous work ("Culinary Class Wars") by introducing a systematic meta-evaluation of the ASH (Authenticity, Sensitivity, Harmony) framework.
In this extended study, we not only evaluate Large Language Models (LLMs) on cuisine transfer tasks but also empirically validate the "LLM-as-a-judge" paradigm itself. We implement and compare eight distinct prompt engineering strategies (ranging from simple scoring to Chain-of-Thought) to identify a practical, human-aligned operating configuration.
- Project Structure
- Setup
- Data Description
- How to Run: Generation & Evaluation
- How to Run: Prompt Engineering Experiments
- Results
- Contributors
- Acknowledgements
The repository is organized into two main parts: data and code, including the new prompt_engineering module.
.
├── data
│ ├── generation
│ │ └── v0_recipes.csv # 4,800 recipes generated by 6 LLMs
│ └── evaluation
│ ├── single # Baseline ASH evaluation outputs
│ │ ├── v0_recipes_eval_4o.csv
│ │ └── v0_recipes_eval_4o_mini.csv
│ ├── 5-round # 5-round evaluation outputs (not included; see Note)
│ └── human # Human-annotated ground truth (Phase 1, 5 annotators x 200 recipes)
│ ├── v0_human_annotation_final_cleaned_H1_USA.csv
│ ├── ... # one file per annotator (H1-H5)
│ └── v0_human_annotation_final_cleaned_human_total.csv
└── code
├── generation # Recipe generation scripts
│ ├── generate_recipes_ollama.py
│ └── generate_recipes_gpt4omini.py
├── evaluation # Standard ASH evaluation scripts
│ ├── single/ # per-evaluator scripts (ollama, 4o, 4o-mini, gemini)
│ └── 5-round/ # 5-round evaluation scripts
└── prompt_engineering # Prompt optimization experiments
└── evaluate_recipes_prompt_check_ollama.py # 8 prompt strategies
- Clone the repository:
git clone https://github.com/dmis-lab/ASH2608.git
cd ASH2608
- Install dependencies:
pip install -r requirements.txt
- API Keys: Ensure you have the necessary API keys configured:
- OpenAI API key:
../API_KEY/API_KEY_openai.txt - Google Gemini API key:
../API_KEY/API_KEY_gemini.txt
- data/generation: Contains 4,800 recipes generated by 6 models across 40 cuisines.
- data/evaluation/single: Baseline ASH evaluation outputs from the GPT-4o and GPT-4o mini evaluators.
- data/evaluation/human: Human-annotated ground truth from the five Phase 1 annotators (200 recipes each), provided per annotator and as a merged file.
- Note: Raw outputs of the prompt-strategy meta-evaluation (153,600 evaluations) and the 5-round evaluation results are not included due to file-size constraints; they can be reproduced with the script below or obtained from the authors upon request.
Generate recipes using the standardized prompt template.
# Open-source models via Ollama
python code/generation/generate_recipes_ollama.py
# GPT-4o mini via the OpenAI API
python code/generation/generate_recipes_gpt4omini.py
Evaluate the generated recipes using the default scoring prompt.
python code/evaluation/5-round/evaluate_recipes_5_ollama.py data/generation/v0_recipes.csv
This section reproduces the meta-evaluation experiments (Table 5 in the paper) to identify the operating prompt strategy.
Run the comprehensive evaluation script. This script utilizes multiprocessing to distribute tasks across available GPUs and evaluates recipes using 8 distinct prompt strategies (Default, Role-Playing, Scoring Scale, CoT, etc.) and multiple evaluator models.
Usage:
Ensure your Ollama server is running and the required models (e.g., gemma2:9b, mistral:7b, llama3.1:8b) are pulled.
# Run the prompt check script
python code/prompt_engineering/evaluate_recipes_prompt_check_ollama.pyExpected Output: Rankings of prompt strategies based on MSE. (e.g., Strategy 3: Scoring Scale Specification typically yields the lowest MSE).
- Generative Capability: Comparison of 6 LLMs showing the trade-off between Sensitivity (Style) and Authenticity (Substance).
- Evaluator Reliability: The "Scoring Scale Specification" strategy achieved the lowest point-estimate MSE (1.087) and clearly outperformed complex Chain-of-Thought prompts, highlighting a "Complexity Paradox" in automated evaluation.
| Name | Affiliation | |
|---|---|---|
| Hoonick Lee (First Author) | Dept. of Computer Science & Engineering, Korea University | hoonick@korea.ac.kr |
| Mogan Gim | Dept. of Biomedical Engineering, Hankuk University of Foreign Studies | gimmogan@hufs.ac.kr |
| Donghyeon Park | Dept. of AI and Data Science, Sejong University | parkdh@sejong.ac.kr |
| Donghee Choi† | School of Computer Science & Engineering, Pusan National University | dchoi@pusan.ac.kr |
| Jaewoo Kang† | Dept. of Computer Science & Engineering, Korea University | kangj@korea.ac.kr |
† Corresponding Authors
If you use this code or data, please cite:
@article{lee2026ash,
title={{ASH}: A Systematic Automated Evaluation Framework for Creative Text Generation in Cuisine Transfer Tasks},
author={Lee, Hoonick and Gim, Mogan and Park, Donghyeon and Choi, Donghee and Kang, Jaewoo},
journal={IEEE Access},
year={2026},
note={to appear}
}The work of Hoonick Lee and Jaewoo Kang was supported by the National Research Foundation of Korea (NRF) under Grant NRF-2023R1A2C3004176. The work of Mogan Gim was supported by Hankuk University of Foreign Studies Research Fund (of 2025). The work of Donghee Choi was supported by a New Faculty Research Grant of Pusan National University, 2025; by BK21 FOUR, Korean Southeast Center for the 4th Industrial Revolution Leader Education; by the Institute of Information & Communications Technology Planning & Evaluation (IITP) under the Artificial Intelligence Convergence Innovation Human Resources Development (IITP-2026-RS-2023-00254177) grant funded by the Korea government (MSIT); and by the Pusan National University-LG Electronics Smart Control Center.