3.9 KiB

Raw Blame History

Vetrag

Lightweight veterinary visit reasoning helper with LLM-assisted keyword extraction and disambiguation.

Features

Switch seamlessly between local Ollama and OpenRouter (OpenAI-compatible) LLM backends by changing environment variables only.
Structured JSON outputs enforced using provider-supported response formats (Ollama format, OpenAI/OpenRouter response_format: { type: json_object }).
Integration tests using mock LLM & DB (no network dependency).
GitHub Actions CI (vet, test, build).

Quick Start

1. Clone & build

git clone <repo-url>
cd vetrag
go build ./...

2. Prepare data

Ensure config.yaml and maindb.yaml / db.yaml exist as provided. Visit data is loaded at runtime (see models.go / db.go).

3. Run with Ollama (local)

Pull or have a model available (example: ollama pull qwen2.5):

export OPENAI_BASE_URL=http://localhost:11434/api/chat
export OPENAI_MODEL=qwen2.5:latest
# API key not required for Ollama
export OPENAI_API_KEY=

go run .

4. Run with OpenRouter

export OPENAI_BASE_URL=https://openrouter.ai/api/v1/chat/completions
export OPENAI_API_KEY=sk-or-XXXXXXXXXXXXXXXX
export OPENAI_MODEL=meta-llama/llama-3.1-70b-instruct  # or any supported model

go run .

Open http://localhost:8080/ in your browser.

5. Health & Chat

curl -s http://localhost:8080/health
curl -s -X POST http://localhost:8080/chat -H 'Content-Type: application/json' -d '{"message":"my dog has diarrhea"}' | jq

Environment Variables

Variable	Purpose	Default (if empty)
OPENAI_BASE_URL	LLM endpoint (Ollama chat or OpenRouter chat completions)	`http://localhost:11434/api/chat`
OPENAI_API_KEY	Bearer token for OpenRouter/OpenAI-style APIs	(unused if empty)
OPENAI_MODEL	Model identifier (Ollama model tag or OpenRouter model slug)	none (must set for remote)

How Backend Selection Works

llm.go auto-detects the style:

If the base URL contains openrouter.ai or /v1/ it uses OpenAI-style request & parses choices[0].message.content.
Otherwise it assumes Ollama and posts to /api/chat with format for structured JSON.

Structured Output

We define a JSON Schema-like map internally and:

Ollama: send as format (native structured output extension).
OpenRouter/OpenAI: send response_format: { type: "json_object" } plus a system instruction describing the expected keys.

Prompts

Prompts in config.yaml have been adjusted to explicitly demand JSON only. This reduces hallucinated prose and plays well with both backends.

Testing

Run:

go test ./...

All tests mock the LLM so no network is required.

CI

GitHub Actions workflow at .github/workflows/ci.yml runs vet, tests, build on push/PR.

Troubleshooting

Symptom	Cause	Fix
Provider error referencing `response_format` and `json_schema`	Some providers reject `json_schema`	We now default to `json_object`; ensure you pulled latest changes.
Empty response	Model returned non-JSON or empty content	Enable debug logs (see below) and inspect raw response.
Non-JSON content (code fences)	Model ignored instruction	Try a stricter system message or switch to a model with better JSON adherence.

Enable Debug Logging

Temporarily edit main.go:

logrus.SetLevel(logrus.DebugLevel)

(You can also refactor later to read a LOG_LEVEL env var.)

Sanitizing Output (Optional Future Improvement)

If some models wrap JSON in text, a post-processor could strip code fences and re-parse. Not implemented yet to keep logic strict.

Next Ideas

Add retry with exponential backoff for transient 5xx.
Add optional json fallback if a provider rejects json_object.
Add streaming support.
Add integration test with recorded OpenRouter fixture.

License

(Choose and add a LICENSE file if planning to open source.)

3.9 KiB Raw Blame History