Human-in-the-Loop AI: Crowdsourcing, LLMs, and Evaluation - 605.734

Human-in-the-Loop AI: Crowdsourcing, LLMs, and Evaluation explores how modern AI systems are built, improved, and audited through the combined strengths of people and machines. As large language models reshape the way organizations collect data, make decisions, and deploy intelligent systems, this course examines when human judgment still matters, when AI can automate complex tasks, and when hybrid human-AI workflows produce the best results. Students will study crowdsourcing, task and prompt design, data quality, inter-annotator agreement, evaluation methods, and bias in both human- and AI-generated outputs. Through hands-on assignments and applied projects, students will compare crowd workers, LLMs, and human-AI pipelines on realistic problems in classification, annotation, conversational systems, and responsible AI evaluation. The course emphasizes practical skills in designing reliable workflows, measuring quality, and identifying failure modes in intelligent systems. By the end of the course, students will be able to build and assess human-in-the-loop AI systems that are effective, scalable, and trustworthy.