Documentation

Welcome to the documentation for Grew-TSE, a tool for the query-based generation of Minimal-Pair Datasets from treebanks for Targeted Syntactic Evaluation of LLMs. The query language of choice is GREW (Graph Rewriting for NLP). You can pronounce this package a bit like the german word Grütze, meaning grits or groats.

Overview

• Introduction
  • What does a “minimal-pair syntactic test” look like?
  • Try out the Dashboard on Hugging Face 🤗
• Installation
  • Opam & Grewpy
  • Grew-TSE Pipeline
  • Evaluation
• Usage
  • Quick Start
  • Parsing a Treebank
  • Defining a GREW Query
  • Generating Datasets
  • Creating Minimal Pairs
  • End-to-End Workflow
• Issues
• About

Below you will find documentation for the Grew-TSE codebase.

The Main Pipeline

• Pipeline
• Preprocessing

Evaluation Tools For Syntactic Performance

• Evaluation
• Metrics

And below some examples…

Examples

• Tutorial: Georgian Intransitive Nom→Erg Minimal-Pair Tests