Package 'ragR'

Description

The ragR package provides a Retrieval-Augmented Generation (RAG) system implemented in R. It includes:

• A data ingestion pipeline for PDF, text, and Word documents

• An R-native vector store for chunk embeddings

• A configurable RAG pipeline that uses OpenAI embeddings and chat models

• Persistent QA logging for reproducible evaluation

• LLM-scored RAGAS-style metrics, including context precision, context recall, answer relevance, and faithfulness

• A Plumber-based HTTP API suitable for chatbot and evaluation frontends

Architecture

The package is organized into several modules:

• Ingestion (ingestion_files.R) Extracts text from PDF/TXT/DOCX files, chunks it, and prepares data for embedding and storage. Ingestion is handled through package functions and development scripts, not through the minimal HTTP API.

• Embeddings & Chat (embeddings_openai.R) Wraps the OpenAI REST API for embeddings, such as "text-embedding-3-small", and chat models, such as "gpt-4o-mini". Reads the API key from OPENAI_API_KEY.

• Vector Store (vectorstore_interface.R) Implements an R-native vector store using tibbles and numeric matrices, with functions to add, query, and clear stored embeddings. Also includes vectorstore_clear_all() for wiping the entire vector store in one operation.

• RAG Pipeline (rag_pipeline.R) Given a user question, the pipeline:

  1. Computes an embedding for the question

  2. Retrieves top-k similar chunks from the vector store

  3. Builds a grounded context-aware prompt

  4. Calls the OpenAI chat model to generate an answer

• QA Logging & Persistence (qa_logging.R, qa_persistence.R) Stores each interaction in db/qa_log.rds, including the question, model answer, retrieved chunks, final grounded prompt, model names, and timestamps. Provides helpers to load, save, and clear logs.

• RAGAS Metrics & Summary (ragas_metrics.R, ragas_summary.R) Computes LLM-scored RAGAS-style metrics, including context precision, context recall, answer relevance, faithfulness, and an overall score. Provides summary helpers for aggregating metric results across QA pairs.

• Reports (ragas_report.R) Generates a performance report by:

  • Computing LLM-scored metrics from the QA log

  • Saving db/qa_metrics.rds

  • Writing reports/ragas/ragas_summary.csv

  • Writing a bar chart of mean metrics to reports/ragas/ragas_means.png

• API Handlers (api_handlers.R) Functions that connect the core logic to HTTP endpoints. These are called by the Plumber router defined in ⁠inst/api/⁠.

HTTP API

When scripts/dev/run_api.R is executed, a Plumber server exposes:

• POST /chat Runs the RAG pipeline for a user question and logs the result into db/qa_log.rds.

• GET /ragas Computes and returns LLM-scored RAGAS-style metrics and a summary for the logged QA pairs.

• POST /ragas/report Computes LLM-scored metrics from the QA log, saves the metrics and report artifacts to disk, and returns a small summary object.

• POST /ragas/clear Clears the QA log, QA metrics, and report files.

• POST /clear Clears one vector-store collection.

• POST /clear_all Clears the entire vector store across all collections. Uses api_clear_all_handler(), which calls vectorstore_clear_all().

Usage

Programmatic usage inside R:

library(ragR)

# Run a RAG query
res <- query_rag(
  question        = "What is this document about?",
  collection      = "default",
  top_k           = 5,
  embedding_model = "text-embedding-3-small",
  chat_model      = "gpt-4o-mini",
  system_prompt   = "You are a helpful assistant."
)

cat(res$answer)

As a backend service:

# From the project root
source("scripts/dev/run_api.R")
# Visit http://127.0.0.1:8000/__docs__/ for interactive docs

Authors

Muhammad Aimal Rehman Zhili Lu Chi-Kuang Yeh

Author(s)

Maintainer: Muhammad Aimal Rehman [email protected]

Authors:

• Zhili Lu

• Chi-Kuang Yeh

Description

This handler receives a parsed JSON request body, runs the RAG pipeline, appends the interaction to the QA log, and returns the model answer.

Usage

api_chat_handler(body)

Arguments

Value

A list suitable for JSON serialization.

Description

This handler wipes the R-native vector store by calling vectorstore_clear_all(), removing all collections and stored embeddings.

Usage

api_clear_all_handler()

Value

A list with elements status and message, suitable for JSON serialization.

Description

This handler deletes one collection from the R-native vector store. If no collection is supplied, "default" is used.

Usage

api_clear_handler(body = NULL)

Arguments

Value

A list with status, collection, and message.

Description

Clears the saved QA log, QA metrics file, and generated RAGAS report files.

Usage

api_ragas_clear_handler(
  qa_log_path = "db/qa_log.rds",
  qa_metrics_path = "db/qa_metrics.rds",
  output_dir = "reports/ragas"
)

Arguments

Value

A list with status and message.

Description

API handler: compute RAGAS metrics and summary from saved QA log

Usage

api_ragas_handler(path = "db/qa_log.rds", judge_model = "gpt-4o-mini")

Arguments

Value

A list with status, n_qa, metrics, summary.

Description

API handler: generate RAGAS performance report

Usage

api_ragas_report_handler(
  qa_log_path = "db/qa_log.rds",
  qa_metrics_path = "db/qa_metrics.rds",
  output_dir = "reports/ragas",
  judge_model = "gpt-4o-mini"
)

Arguments

Value

A list with status, n_qa, qa_metrics_path, summary_csv_path, and plot_path.

Description

Splits text into individual sentences. Each returned element is intended to be exactly one sentence (best-effort based on punctuation).

Usage

chunk_text_sentence(text)

Arguments

Value

Character vector; one sentence per element.

Description

This helper overwrites the QA log file with an empty QA log, as created by qa_log_empty(). It is intended for starting a fresh evaluation session.

Usage

clear_qa_log(path = "db/qa_log.rds")

Arguments

Value

Invisibly, the path to the saved file.

Description

This helper overwrites the QA metrics file with an empty metrics tibble, as created by qa_metrics_empty().

Usage

clear_qa_metrics(path = "db/qa_metrics.rds")

Arguments

Value

Invisibly, the path to the saved file.

Description

Convenience wrapper for computing LLM-scored RAGAS-style metrics.

Usage

compute_ragas_metrics(
  qa_log,
  judge_model = "gpt-4o-mini",
  seed = NULL,
  embedding_model = "text-embedding-3-small",
  answer_relevance_strictness = 3L
)

Arguments

Value

A metrics tibble.

Description

This implementation mirrors the structured Python RAGAS workflow rather than asking the judge model for one direct scalar score per metric.

Usage

compute_ragas_metrics_llm(
  qa_log,
  judge_model = "gpt-4o-mini",
  seed = NULL,
  embedding_model = "text-embedding-3-small",
  answer_relevance_strictness = 3L
)

Arguments

Details

Notes:

• context_precision uses answer_reference if available; otherwise it uses answer_model, matching the Python with-reference / without-reference split.

• answer_relevance requires an embedding helper to be available in the package environment.

Value

A tibble with one row per qa_id and metric columns.

Description

Used by the RAG pipeline and RAGAS-style metric prompts to generate text from a constructed prompt.

Usage

generate_openai_chat(
  prompt,
  model = "gpt-4o-mini",
  system_message = NULL,
  temperature = 0,
  max_output_tokens = 512L,
  seed = NULL,
  api_key = NULL
)

Arguments

Value

Character scalar containing the model's answer.

Description

Loads a QA log from disk, computes LLM-scored RAGAS-style metrics, and writes:

• metrics RDS (qa_metrics_path)

• summary CSV (ragas_summary.csv)

• means bar plot PNG (ragas_means.png)

Usage

generate_ragas_report(
  qa_log_path = "db/qa_log.rds",
  qa_metrics_path = "db/qa_metrics.rds",
  output_dir = "reports/ragas",
  judge_model = "gpt-4o-mini"
)

Arguments

Value

A list with: status, n_qa, qa_metrics_path, summary_csv_path, plot_path.

Description

This is an explicit alias for generate_ragas_report().

Usage

generate_ragas_report_llm(
  qa_log_path = "db/qa_log.rds",
  qa_metrics_path = "db/qa_metrics.rds",
  output_dir = "reports/ragas",
  judge_model = "gpt-4o-mini"
)

Arguments

Value

A list with: status, n_qa, qa_metrics_path, summary_csv_path, plot_path.

Description

Calls the OpenAI embeddings endpoint to obtain vector representations for one or more input texts. Used by both ingestion and query-time retrieval.

Usage

get_openai_embeddings(texts, model = "text-embedding-3-small", api_key = NULL)

Arguments

Value

A numeric matrix with one row per input text.

Description

Reads PDF, DOCX, or TXT files, extracts text, cleans it lightly, chunks it, embeds chunks, and stores them in a vector store collection.

Usage

ingest_documents(
  paths,
  collection = "default",
  chunk_size = 500L,
  chunk_overlap = 50L,
  chunking_strategy = c("character", "sentence"),
  embedding_model = "text-embedding-3-small",
  embedding_batch_size = 128L,
  embedding_max_chars = 8000L,
  retry = TRUE,
  max_retries = 5L,
  resume = TRUE,
  checkpoint_path = NULL,
  use_openai = TRUE,
  verbose = TRUE
)

Arguments

Details

Chunking strategies:

• "character": overlapping fixed-size character windows.

• "sentence": strict sentence splitting via chunk_text_sentence().

Cleaning strategy (minimal, deterministic):

• Replace carriage returns and newlines with spaces

• Remove control characters

• Collapse multiple whitespace to a single space

• Trim leading/trailing whitespace

Embedding strategy (robust by default):

• Embeddings are computed in batches (default embedding_batch_size = 128)

• Batches are retried with exponential backoff on transient failures (e.g., 429/5xx)

• If a 400 occurs due to request size, the batch is automatically split

• Each chunk is truncated to a safe maximum length before embedding

• Optional resumable ingestion via a local checkpoint file

Value

A tibble with a per-file ingestion summary (collection, path, n_chunks).

Description

This helper loads a QA log tibble from an RDS file. If the file does not exist, it returns an empty QA log created by qa_log_empty().

Usage

load_qa_log(path = "db/qa_log.rds")

Arguments

Value

A tibble containing the QA log.

Description

This helper loads QA metrics from an RDS file. If the file does not exist, it returns an empty metrics tibble created by qa_metrics_empty().

Usage

load_qa_metrics(path = "db/qa_metrics.rds")

Arguments

Value

A tibble containing the QA metrics.

Description

Placeholder for a configuration loader. In the future, this can read a YAML file and merge it with environment variables and defaults.

Usage

load_rag_config(path = NULL)

Arguments

Value

A list of configuration values. Currently returns an empty list.

Description

Adds one row to an in-memory QA log tibble.

Usage

log_rag_interaction(
  qa_log,
  question,
  rag_result,
  collection,
  chat_model = "gpt-4o-mini",
  embedding_model = "text-embedding-3-small",
  qa_id = NULL,
  timestamp = Sys.time()
)

Arguments

Value

Updated QA log tibble.

Description

Creates a simple bar chart of mean metric values.

Usage

plot_ragas_means(summary_df, output_path, width = 1200, height = 700)

Arguments

Value

Invisibly, output_path.

Description

Standard schema for storing Q/A interactions produced by the RAG pipeline.

Usage

qa_log_empty()

Value

A tibble with zero rows and the standard QA log columns.

Description

This helper creates an empty tibble with the columns used to store RAGAS-style evaluation metrics for each QA interaction.

Usage

qa_metrics_empty()

Details

All metric values are expected to be in $[0,1]$.

Value

A tibble with zero rows and the standard QA metrics columns.

Description

Takes a user question, embeds it, retrieves relevant chunks from a vector store, constructs a grounded RAG prompt, and calls an LLM to generate an answer.

Usage

query_rag(
  question,
  collection = "default",
  top_k = 4L,
  embedding_model = "text-embedding-3-small",
  chat_model = "gpt-4o-mini",
  temperature = 0,
  max_output_tokens = NULL,
  score_threshold = 0,
  system_prompt = ""
)

Arguments

Value

A list with:

Description

This helper saves a QA log tibble, as produced by log_rag_interaction(), to an RDS file. By default, it writes to db/qa_log.rds inside the project directory.

Usage

save_qa_log(qa_log, path = "db/qa_log.rds")

Arguments

Value

Invisibly, the path to the saved file.

Description

This helper saves a QA metrics tibble, as produced by compute_ragas_metrics(), to an RDS file. By default, it writes to db/qa_metrics.rds inside the project directory.

Usage

save_qa_metrics(qa_metrics, path = "db/qa_metrics.rds")

Arguments

Value

Invisibly, the path to the saved file.

Description

Computes mean, standard deviation, minimum, and maximum for each metric column in a QA-metrics tibble.

Usage

summarize_ragas(qa_metrics)

Arguments

Value

A tibble with columns: metric, mean, sd, min, max.

Description

Completely wipes the vector store by replacing it with an empty tibble.

Usage

vectorstore_clear_all()

Value

Invisibly, TRUE.

Description

Delete a collection from the R-native vector store

Usage

vectorstore_delete_collection(collection)

Arguments

Value

Invisibly, TRUE on success.

Description

This function loads the current contents of the R-native vector store from db/vectorstore.rds. If the file does not exist yet, it returns an empty tibble with the expected columns: collection, id, text, embedding, and metadata.

Usage

vectorstore_load()

Value

A tibble with one row per stored chunk, or zero rows if the store is empty.

Description

Query the R-native vector store for nearest neighbors

Usage

vectorstore_query(collection, query_embedding, top_k = 4L)

Arguments

Value

A tibble with columns collection, id, text, score, and metadata.

Description

Upsert embeddings into the R-native vector store

Usage

vectorstore_upsert(collection, ids, embeddings, documents, metadatas = NULL)

Arguments

Value

Invisibly, TRUE on success.