lightonai/Reason-ModernColBERT

Reason-ModernColBERT is a late interaction model trained on the reasonir-hq dataset. It achieves extremely competitive performance on the BRIGHT benchmark aimed at evaluating reasoning-intensive retrieval performance, outperforming all existing models up to 7B (more than 45 times its size) and ev...

Architecture

ModernBERT

Parameters

149M

Tasks

Encode

Outputs

Multi-Vec

Dimensions

Multi-Vec: 128

Max Sequence Length

8,192 tokens

License

cc-by-nc-4.0

Languages

View on HuggingFace → Fine-tuned from lightonai/GTE-ModernColBERT-v1

Benchmarks

CQADupstackPhysicsRetrieval

scientific retrieval en

Duplicate question retrieval from StackExchange Physics

Corpus: 38,314 Queries: 1,039

Quality

ndcg at 10 0.4382

map at 10 0.3824

mrr at 10 0.4397

Performance L4 b1 c16

Corpus TPS 29.6K

Corpus p50 61.0ms

Query TPS 2.9K

Query p50 57.8ms

Reference →

CosQA

technology retrieval en

Code search with natural language queries

Corpus: 6,267 Queries: 500

Quality

ndcg at 10 0.3443

map at 10 0.2641

mrr at 10 0.2620

Performance L4 b1 c16

Corpus TPS 12.7K

Corpus p50 60.8ms

Query TPS 1.6K

Query p50 59.1ms

Reference →

FiQA2018

finance retrieval en

Financial opinion mining and question answering

Corpus: 57,599 Queries: 648

Quality

ndcg at 10 0.4081

map at 10 0.3315

mrr at 10 0.4936

Performance L4 b1 c16

Corpus TPS 30.7K

Corpus p50 71.5ms

Query TPS 3.0K

Query p50 59.6ms

Reference →

LegalBenchConsumerContractsQA

legal retrieval en

Question answering on consumer contracts

Corpus: 153 Queries: 396

Quality

ndcg at 10 0.7777

map at 10 0.7287

mrr at 10 0.7307

Performance L4 b1 c16

Corpus TPS 50.0K

Corpus p50 144.2ms

Query TPS 4.2K

Query p50 60.7ms

Reference →

NFCorpus

medical retrieval en

Biomedical literature search from NutritionFacts.org

Corpus: 3,593 Queries: 323

Quality

ndcg at 10 0.3580

map at 10 0.1342

mrr at 10 0.5702

Performance L4 b1 c16

Corpus TPS 44.6K

Corpus p50 89.2ms

Query TPS 1.4K

Query p50 56.3ms

Reference →

SCIDOCS

scientific retrieval en

Citation prediction, document classification, and recommendation for scientific papers

Corpus: 25,656 Queries: 1,000

Quality

ndcg at 10 0.1710

map at 10 0.0986

mrr at 10 0.2991

Performance L4 b1 c16

Corpus TPS 27.0K

Corpus p50 80.8ms

Query TPS 2.8K

Query p50 60.8ms

Reference →

SciFact

scientific retrieval en

Scientific claim verification using research literature

Corpus: 5,183 Queries: 300

Quality

ndcg at 10 0.7380

map at 10 0.6953

mrr at 10 0.7076

Performance L4 b1 c16

Corpus TPS 41.6K

Corpus p50 83.7ms

Query TPS 4.1K

Query p50 60.4ms

Reference →

StackOverflowQA

technology retrieval en

Programming question answering from Stack Overflow

Corpus: 19,931 Queries: 1,994

Quality

ndcg at 10 0.5934

map at 10 0.5591

mrr at 10 0.5591

Performance L4 b1 c16

Corpus TPS 35.4K

Corpus p50 84.4ms

Query TPS 62.1K

Query p50 75.3ms

Reference →

Benchmarks

CQADupstackPhysicsRetrieval

CosQA

FiQA2018

LegalBenchConsumerContractsQA

NFCorpus

SCIDOCS

SciFact

StackOverflowQA

Self-hosted inference for search & document processing