Article URL: https://github.com/Center-for-AI-Innovation/ai-teaching-assistant-uiuc Comments URL: https://news.ycombinator.com/item?id=48942721 Points: 16 # Comments: 0

Our system runs 11 separate models in parallel for text/image retrieval, generation, moderation and ranking and still achieves a median 2-second response time. We use data from textbooks, lecture videos, and student QA forums (ordered, subjuctively, by importance). None of this data is currently available publically because this project was not granted those rights by the authors. My favorite contribution is the novel approach of semantic search retrieval during RLHF, using a dataset I iteratively produced by hiring a team of five Electrical Engineering students. That data is freely available on Huggingface here: https://huggingface.co/datasets/kastan/rlhf-qa-comparisons. We specifically cover the material necessary in the UIUC course ECE 120, intro to Electrical Engineering. We have a lot of models here. To evaluate which ones are helping, and which hurt, every time we push a new feature we re-run this evaluation. Our evaluation dataset of QA pairs is produced in house, written by expert electrical engineers. Using these questions, we generate answers with each of our models. Finally, we ask GPT-3 if the generated are "better" or "worse" than the ground truth answers written by humans. One limitation is that GPT-3 evaluates itself. GPT-3 nearly always thinks that GPT-3 is great, which is probably not true and a limitation of this evaluation method. Maybe we should run this same evaluation with Cohere's models to compare. Nevertheless, iterative evaluation was crucial to ensure our new features were making our system better. This project is fully open source, with the exception of commercial textbooks. I highly encourage you to simply plug in your own Pinecone database of documents and use this in your work! We rely on these APIs enumerated in run_ta_gradio.sh. Be sure to add your own 😄 Simply build your own Pinecone database of your documents. We open source the scripts we use for this, where our source material is in PDF or plaintext and our lecture slides are .jpg images (sneakily exported from .pptx).