Building UTSA GPT: Teaching Gemini to Answer From My Data, Not Its Own

Why This Project Exists

Large language models are very good at sounding confident, but very bad at admitting when they don't know something. If you ask Gemini what time your favorite sports team plays this weekend, it may struggle to answer honestly. That information isn't in its training data, and it can't always go look it up. Thus, it produces a sentence with the shape of a correct answer and it hopes for the best.

One fix for this is known as Retrieval-Augmented Generation, or RAG. The idea is that instead of asking the model to know things, you hand it the relevant documents at the moment you ask the question, then tell it to answer using only what's in front of it. The model is no longer in charge of providing information and instead acts as a reasoning engine for your data.

My team wanted hands-on experience with the whole thing. We didn't want to build a glorified API wrapper. We wanted to build the scraper, the ingestion, the embedding cache, the retriever, the reranker, and the service layer. Our mentor for the project taught us how to go about planning our project, and splitting responsibilities. One of my partners was much more interested in the React side, another interested in databases and storage, and another interested in cyber security and cloud. Our split was very clean and organized: I owned everything behind a single POST /chat endpoint, and our frontend partner owned everything in front of it. I was able to work on the backend without worrying about the frontend, only worrying about the request and response shape for that endpoint. My other two partners worked on ingestion and security, allowing me to focus on building the service layer upon their data.

We used an NFL schedule as our first test dataset. It was easy to reason about, easy to verify answers against, and small enough that we could eyeball whether retrieval was working. But the backend was designed from the beginning to be collection-agnostic. Point it at a different Firestore collection and you have a chatbot for a different dataset, without touching the retrieval code. We would later add a database covering over 40 data points on 1000 different computers sold at Best Buy, and the same backend worked without modification.

What RAG Actually Is

When a user asks a question, we don't send that question straight to the LLM. First we search our own data for a handful of 'documents' that look relevant. Then we paste those documents into the prompt alongside the question and instruct the model to answer using only that context.

The strength and weakness of RAG is that the quality of your chatbot is almost entirely determined by the quality of your search. If you hand the model the wrong five documents, it will either hallucinate or tell you an answer doesn't exist. However, with the right five quality documents, even a simple, inexpensive model gives you a precise answer.

The pipeline I ended up with has six stages:

Data flows one direction: Data Source (ESPN) → CSV → Firestore → LlamaIndex Documents → embeddings + BM25 index → hybrid retrieval → cross-encoder rerank → Gemini → response.

The Tech Stack

A few of these choices shaped everything downstream, so they're worth explaining before I walk through the code.

Embeddings run locally. I could have called an embedding API for every document and every query. Instead all-MiniLM-L6-v2 runs on CPU, in milliseconds, for free, and never sends our data to a third party. For short factual documents like schedule rows, the quality difference against a paid embedding API was minimal.

Firestore was a team decision. We were already working within the google ecosystem, and the document model worked if we assumed "one row equals one chunk of context."

Step 1: Getting the Data

Before there was anything to retrieve, there had to be data. The NFL schedule is sorted into weeks, so my partner wrote a Selenium scraper that walks through each game week and pulls each table.

def scrape_current_page(driver, all_games, clean_games):
    days = driver.find_elements(By.CLASS_NAME, "Table__Title")
    tables = driver.find_elements(By.CLASS_NAME, "Table__TBODY")

    for i in range(len(days)):
        date = days[i].text
        rows = tables[i].find_elements(By.TAG_NAME, "tr")

        for row in rows:
            cols = row.find_elements(By.TAG_NAME, "td")
            if len(cols) >= 2:
                all_games.append({
                    "Date": date,
                    "Away": cols[0].text,
                    "Home": cols[1].text,
                })
    clean_data(all_games, clean_games)

ESPN gives you "Sunday, September 7, 2025" in one cell and "@ Philadelphia" in another, so there's a pass that splits the day off the date, strips the year, and removes the @ from the home team. The result was a CSV of 47 games, which then got pushed into Firestore.

Step 2: Ingestion

The job here is to take a Firestore collection and turn it into a list of Document objects the rest of the pipeline understands.

def load_documents(collection: str) -> list[Document]:
    db = _get_db()
    documents = []
    for doc in db.collection(collection).stream():
        text = ", ".join(f"{k}: {v}" for k, v in doc.to_dict().items())
        documents.append(Document(text=text, metadata={"doc_id": doc.id}))
    return documents

A schedule row comes out the other side as a single document: "day: Thursday, date: September 4, away: Dallas, home: Philadelphia".

Why serialize generically like that? Because I initially wrote NFL field names into this function, the backend was specific to NFL data. The generic key: value format extracts the data into a format the embedding model can understand, meaning this ingestion never has to change when we point the chatbot at a new collection. It would be easy to customize this step for individual datasets, but this approach ensured we could reuse the retrieval and generation layers without worrying about what data we were receiving.

The _get_db() helper initializes Firebase Admin exactly once, working as a lazy singleton initializer, and checks firebase_admin._apps before doing it.

Step 3: Embedding, and a Cache That Doesn't Lie

Every document has to become a vector before you can do similarity search. Embedding 48 documents on CPU doesn't take long. However, this quickly builds as you introduce more documents, and the embedding model was the slowest part of the pipeline. I obviously didn't want to re-embed everything every time I restarted the server, so I built a vector cache. The first version keyed on file modification time, which seemed like the obvious thing to do, but it cost me a lot of time.

Modification times happened more often than I was expecting. A git checkout touches files without changing them. Some editors rewrite a file on save even when nothing changed. I switched to a different IDE partway through the semester and suddenly every restart was re-embedding every document, and it took me a very long time to connect the two.

The fix was to stop asking when the documents changed and just check if they changed. I hash the contents of every document and store that hash alongside the vectors. If the hash matches, I know the documents are identical and I can skip embedding.

def _hash_contents(contents: list[str]) -> str:
    combined = "\n".join(contents).encode("utf-8")
    return hashlib.sha256(combined).hexdigest()


def load_or_embed(agent, contents):
    current_hash = _hash_contents(contents)

    if CACHE_FILE.exists():
        cached = json.loads(CACHE_FILE.read_text())
        if cached.get("hash") == current_hash:
            agent._build_bm25(contents)
            return np.array(cached["vectors"])
        print("Documents changed — re-embedding...")

    vectors = agent.embed(contents)
    CACHE_FILE.write_text(json.dumps({"hash": current_hash, "vectors": vectors}))
    return np.array(vectors)

The content hash saved a lot of time. Change one character in one document and the hash changes and we re-embed. Change nothing and we read 48 vectors off disk instantly.

Step 4: Hybrid Retrieval

The default approach to retrieval is to embed the query, run cosine similarity against every document vector, and take the top few. I did that first. It worked well for vague, semantic questions like "tell me about the Cowboys." However, it fell apart on specific ones.

If I were to ask "when do the Cowboys play the Eagles?" a dense-only retriever may prefer a document about Cowboys vs Giants, because in embedding space a document about "Cowboys vs Giants" and a document about "Cowboys vs Eagles" have very similar meaning. Dense embeddings capture meaning, and their weakness is that they can pass over the exact token you actually cared about.

BM25, the classic keyword-matching algorithm that search engines used for decades, has the opposite issue. It nails exact matches and completely misses synonyms and paraphrases.

Using the two retrievers together greatly improves performance and accuracy.

def retrieve(self, query, documents, vectors, top_k=5):
    # Dense scores
    query_vector = np.array(self._embed_model.encode([query])[0])
    cosine_scores = _cosine_similarity(np.array(vectors), query_vector)

    # Sparse scores
    if self._bm25 is not None:
        bm25_scores = np.array(self._bm25.get_scores(query.lower().split()))

        def normalize(s):
            r = s.max() - s.min()
            return (s - s.min()) / (r if r > 0 else 1)

        fused = 0.5 * normalize(cosine_scores) + 0.5 * normalize(bm25_scores)
    else:
        fused = cosine_scores  # fall back to pure vector search

    top_indices = np.argsort(fused)[::-1][:top_k]
    return [documents[i] for i in top_indices]

Here is what the sparse half actually scores on my real retrieval setup. Click through the three modes. The third is a fix I only found while writing this post!

Real BM25 output against all 48 documents. The correct answer for "when do the Cowboys play the Eagles?" is the September 4 game.

The normalization is doing more work than it looks like. Cosine similarity lives roughly in the range 0 to 1. BM25 scores are unbounded and routinely hit 20 or 30. If you add them together raw, you end up with a BM25 retriever that has a small amount of cosine-shaped noise added to it. Min-max normalizing each score array into [0, 1] before fusing is what makes the 50/50 weight actually mean 50/50.

Why 50/50? Because I tried a handful of splits and this one behaved best on the questions we cared about. There's a more principled technique called Reciprocal Rank Fusion that I'd like to try next, but for 48 documents a weighted average was defensible and made it easier to reason about what it was doing.

Step 5: Reranking

After fusion I have a top 10. That's better than a top 10 from either signal alone, but those scores came from a bi-encoder: the model produced one vector for the query and one vector for each document, entirely independently, and then we compared them. That independence is what makes it fast enough to run against an entire database, but it makes it imprecise. The document's vector was computed without ever having seen the query, making them less comparable than they could be. A bi-encoder is a good first pass, but it can be improved.

A cross-encoder takes the query and one document together as a single input and scores that pair jointly. It's dramatically more accurate and dramatically slower, because you have to run the model once per pair instead of reusing precomputed vectors.

It's far too slow to run against every document, but fine when you only have 10 candidates. So I take the top 10 from hybrid retrieval and pass them to a cross-encoder reranker, which produces a final top 5.

def rerank(self, query: str, documents: list[str], top_k: int = 5) -> list[str]:
    pairs = [(query, doc) for doc in documents]
    scores = self._reranker.predict(pairs)
    top_indices = np.argsort(scores)[::-1][:top_k]
    return [documents[i] for i in top_indices]

48 documents in, 10 survive hybrid retrieval, 5 reach the model.

This two-stage shape, something cheap that runs over everything followed by something expensive that runs over the survivors, is one of the most common patterns in production search. It's the same structure as a candidate generator feeding a ranker in a recommender system. I had a hard time understanding how the two approaches were different, but implementing both of them allowed me to witness the difference in cost and accuracy first-hand.

Step 6: Query Rewriting

There's one more stage that sits before retrieval.

We want people to ask questions conversationally. "Who do the Cowboys play this week?" is a perfectly clear question for a human but it's a confusing search query. So before retrieving anything, we send the question to Gemini and ask it to rewrite it into something keyword-dense.

The rewrite prompt is loaded from a plain text file rather than hardcoded, so it can be tuned per dataset without touching Python. The NFL version includes rules like expanding nicknames to full team names and making implicit dates explicit. This step allows us to answer more ambigous questions like "When do the Cowboys play this week?" or "Who plays the Eagles on September 4?" without having to write a more complex retriever.

We tried a couple different prompt templates but settled on one with this ending:

- Make your best effort to pick out specific teams and dates that could fit
  the user's question (e.g. "Blue Star Team" → "Dallas Cowboys")
- Make your best guess to fit a team to the user's requests and profile.
  (e.g. Lives in Texas → "Dallas Cowboys")

This part is doing more than polish. If you look back at the first mode in the retrieval widget above, the raw user question scores zero against all 48 documents. The database stores city names, so the corpus contains Dallas and Philadelphia and never the words "Cowboys" or "Eagles." Without the rewrite expanding nicknames into cities, the entire sparse half of my hybrid retriever contributes nothing at all.

That second rule improves retrieval for the user. The system is using keywords provided by the initial query to give better recommendations. You can ask the model to recommend an NFL game as a fan in Texas, the rewriter will know to ask for the Cowboys or the Texans. The rewriting step helps improve accuracy but also allows those without much knowledge of the database to get more accurate answers with vague questions.

I pass both versions of the question through to the final prompt so the model retrieves on the rewritten query but can answer with the context of the question that was actually asked:

prompt_question = ("Original question: " + question +
                   "\nRewritten query: " + rewritten)

Step 7: The Service Layer

The FastAPI app is deliberately boring. All of the heavy lifting happens once, at startup, inside a lifespan context manager:

@asynccontextmanager
async def lifespan(app: FastAPI):
    collection = os.getenv("FIRESTORE_COLLECTION", "game")

    template_path = Path(__file__).parent / "nfl_query_rewrite.txt"
    template = str(template_path) if template_path.is_file() else None

    agent = get_agent(query_rewrite_template=template)

    docs    = await asyncio.to_thread(load_documents, collection)
    texts   = [d.text for d in docs]
    vectors = await asyncio.to_thread(load_or_embed, agent, texts)

    app.state.agent   = agent
    app.state.docs    = texts
    app.state.vectors = vectors

    yield

The agent is a module-level singleton behind get_agent(). Loading SentenceTransformer and CrossEncoder takes several seconds, and constructing a new agent per request would make every user pay that cost. Loading them at startup means the first real request is fast.

The documents and vectors live on app.state rather than inside the agent, and get passed into answer() on each call. The agent ends up stateless with respect to the corpus, so nothing about it assumes there's only ever one dataset.

Then there are exactly two endpoints, and the interesting one is four lines:

@app.post("/chat", response_model=ChatResponse, tags=["Chat"])
async def chat(body: ChatRequest, request: Request):
    answer, sources = await request.app.state.agent.answer(
        body.question,
        request.app.state.docs,
        request.app.state.vectors,
    )
    return ChatResponse(answer=answer)

That's the entire function my partners and I had to build against. One POST with one string in and one string out. Since the interface was small and settled early on, my partners built the whole frontend in parallel without ever being blocked by me. I was confused about how we would coordinate building two parts of a system in parallel, but it turned out to be easy. We just had to agree on the request and response shape in advance and everything fit perfectly when we integrated.

What Broke Along the Way

The version above works. Several versions before it did not, and most of them failed in a very silent and frustrating manner.

Sync calls inside an async endpoint

When I first wired up FastAPI, I called Gemini directly inside the async handler. Under any concurrency at all, the server would freeze for seconds at a time. The event loop was sitting there waiting on a network response, unable to do anything else.

The Gemini SDK, SentenceTransformer, and CrossEncoder are all synchronous. There are four blocking calls in answer(), and every one of them now gets pushed onto a thread pool:

rewritten  = await self.rewrite_query(question)
candidates = await asyncio.to_thread(self.retrieve, rewritten, documents, vectors, 10)
relevant   = await asyncio.to_thread(self.rerank, rewritten, candidates, 5)
result     = await asyncio.to_thread(
    self._client.models.generate_content, model=_GEMINI_MODEL, contents=prompt
)

async def on a function that then blocks isn't actually asynchronous, and it was worse than plain synchronous code because it seemed correct every time I looked at it.

A comma that broke half my retriever

This is the one I found while writing this post, and it's the best example of the pattern I have.

My documents are serialized as "day: Thursday, date: September 4, away: Dallas, home: Philadelphia". My BM25 index is built by splitting on whitespace:

tokenized = [t.lower().split() for t in texts]

Splitting that document on whitespace produces this:

['day:', 'thursday,', 'date:', 'september', '4,', 'away:', 'dallas,', 'home:', 'philadelphia']

Every token carries the punctuation that followed it. dallas, with a comma is a completely different string from dallas, so a query containing "dallas" cannot match a document where Dallas is the away team. It can only match a document where Dallas happens to be the home team, because that's the last field and has no trailing comma.

The measured effect on the query in the widget above: only 3 of 48 documents scored above zero, and the top-ranked result was the wrong game, a September 14 matchup that merely has Dallas at home. The correct September 4 game came second, and it only scored at all because Philadelphia was its final field.

The fix is one line:

tokenized = [re.findall(r"[a-z0-9]+", t.lower()) for t in texts]

With that, the correct game moves to first with nearly double the score (5.761 vs 3.020), and all 48 documents participate in ranking instead of 3.

This was an interesting bug because nothing was ever visibly broken. The cross-encoder reranker sits downstream and is good enough that it often reordered the top 10 into something sensible anyway. So the system produced good answers, most of the time, while one of its two retrieval signals was operating at a fraction of its capability. I had no eval harness, so I had no way to see it. I found it by printing scores for a blog post.

A prompt path that silently didn't load

This issue was similar to my last bug. The prompt template for the RAG step is loaded from a text file, so it can be tuned without touching Python.

The custom answer prompt was loaded like this, at import time:

_CUSTOM_PROMPT_PATH = "./custom_prompt.txt"

if os.path.exists(_CUSTOM_PROMPT_PATH):
    ...
else:
    _RAG_PROMPT_TEMPLATE = """..."""  # generic fallback

That's a relative path, so it resolves against whatever directory you happened to launch uvicorn from. Start the server from backend/ and you get the carefully tuned NFL prompt, the one with the formatting rules and the instruction to list all of Sunday's games instead of just one. Start it from the repository root and os.path.exists returns False, you silently get the generic fallback, and the only symptom is that answers are a bit worse than they were yesterday.

There was no error and no warning, just quietly degraded output. The query rewrite template in main.py had already been fixed to anchor against Path(__file__).parent; this one hadn't. It now does, and the fallback branch logs a warning instead of saying nothing:

_CUSTOM_PROMPT_PATH = Path(__file__).parent / "custom_prompt.txt"

if _CUSTOM_PROMPT_PATH.is_file():
    _RAG_PROMPT_TEMPLATE = _CUSTOM_PROMPT_PATH.read_text()
else:
    _log.warning("custom_prompt.txt not found at %s — falling back to the "
                 "generic prompt template.", _CUSTOM_PROMPT_PATH)

Honest Limitations

Things that this could do better, that I'd want fixed before real users touched it.

There's no chunking strategy. One Firestore row equals one document. That's fine for schedule rows that are twenty words long. Point this at articles, PDFs, or meeting transcripts and it starts to struggle, because a single "document" won't fit sensibly in a prompt and there's no logic to split long text into overlapping chunks.

The cache is one JSON file. At 48 documents this is completely fine. At 48,000 it is not, because the whole thing is read into memory and every vector is compared on every query. A real system uses FAISS or pgvector and stops doing linear scans.

It would have been nice to test our chatbot against a larger dataset. The NFL schedule is small enough that I could eyeball the results and see whether retrieval was working. It was also small enough that I could run the whole thing on CPU. A larger dataset would have been more realistic, and it would have been nice to see how the system performed under load. Our second dataset, the Best Buy computer database, was larger but still small enough to run on CPU. I would have liked to see how the system performed with a few hundred thousand documents, and whether retrieval quality held up.

Next Steps

An eval set, before anything else. Everything below is guesswork until I can measure whether a change helped, and the tokenizer bug is proof that eyeballing it isn't enough. I needed a set of questions with known answers, and a way to measure whether the system is returning the right answer.

Streaming responses. Gemini supports streaming and FastAPI has StreamingResponse. Right now the user watches a spinner for a couple of seconds while the full answer generates. Tokens appearing as they're produced would make the thing feel twice as fast without being any faster.

A real vector store. FAISS or pgvector, so the index isn't bounded by what fits in a JSON file.

Reciprocal Rank Fusion, to replace the weighted average with something I can justify from more than experimentation.

Takeaways as a Backend Developer

A few things I'd tell myself at the start of this.

The bugs that cost me the most time never threw an exception. The mtime cache, the blocking calls in async handlers, the prompt path that resolved against the wrong directory, the dropped sources field, the commas in the BM25 index. Every one of them ran to completion and returned something plausible. I had to look deeper into the results that were coming back to find them. I should have been printing scores and inspecting the retrieval results from the start, but I wasn't. I was too busy looking at the final answer and assuming that if it looked good, it was good.

Write-ups find bugs. I found the tokenizer bug because I wanted a chart for this post and had to print real numbers to make one. Explaining a system forces you to check whether it does what you've been saying it does. I'd been describing the retriever as hybrid for months while half of it was barely functioning.

Boring engineering is most of the work. The content hash, the to_thread wrappers, the singleton agent, the lifespan hook. None of these are clever. Each one fixed a real bug or removed real latency. Lots of our code worked in a controlled demo environment but failed in production. The principles I learned in Secure Software Engineering really paid off in improving the reliability of our system.

A small interface is a force multiplier. We had one endpoint, one field in, one field out. We made sure that we agreed on the endpoint early. My partners never waited on me and I never waited on them. On a team project with a hard deadline, that mattered a lot.

Thanks for reading. I hope to keep working on this project in the future. If you're building something similar, or you have any recommendations, I'd like to hear about it!

Source: Project