textUpstash/wikipedia-2024-06-bge-m3wikipediaembeddingsbge-m3multilingualragsemantic-searchvector-searchparquetretrieval

Wikipedia Embeddings BGE-M3 Multilingual

June 2024

Free

Open dataset

Sample structure: 100 / 100
3 download links issued
Seller: DataBazaar
Sign up to download

Already have an account? Log in

Agent? Connect your account →

Category
Text
Records
11,800,000 rows
Format
PARQUET
Update Frequency
Not documented
Collection Method
auto_imported_huggingface_federated
PII
No flagged field names; not a privacy audit
File Size
~50669.9 MB
Download links issued
3

Source, license and coverage

Supplier documentation. These claims are separate from the automated sample score. A listing edit date is not a data freshness date.

License
apache-2.0
Source / creator
huggingface: Upstash/wikipedia-2024-06-bge-m3
Collection method
auto_imported_huggingface_federated
Coverage / snapshot
June 2024
Data last updated
Not documented
Update schedule
Not documented

Source documentation ↗

License terms ↗

Sample structure score: 100 / 100

This automated check describes the inspected sample, not factual accuracy, legal rights, representativeness, or the quality of the entire dataset. It is not a customer rating.

Assessed 10 sample records (JSON) on 2026-10-09. All records in the provided sample were checked.

CheckPointsEvidence
Populated cells50 / 5050 of 50 top-level cells contain a value. Null, absent and blank values count as missing; zero and false count as populated.
Consistent value types30 / 3050 of 50 populated cells match their column's most common observed type. Types are inferred, not checked against real-world truth.
Consistent record shape20 / 2010 of 10 records have the expected fields. CSV/TSV use the header width; JSON uses the union of observed keys.
Field-level findings and improvements

Check missing cells and mixed types below. Document intentional missing values or mixed types in your field descriptions. Do not fill legitimate unknowns with invented values just to increase this score.

FieldMissing cellsMost common typeOther populated types
id0 / 10string0 / 10
url0 / 10string0 / 10
title0 / 10string0 / 10
text0 / 10string0 / 10
embedding0 / 10object0 / 10
How the score is calculated, its limitations, and how to correct an assessment →

About this data

Paragraph embeddings from Wikipedia across 11 languages, generated with the multilingual BGE-M3 model. Suitable for multilingual semantic search, retrieval-augmented generation, and retrieval evaluation tasks.

Retrieve with your agent or Python

Create an account and configure DATABAZAAR_API_KEY. This example retrieves free or already purchased data; it never makes a purchase. For a multi-file dataset, choose a file index from the manifest.

Download the Python example
python3 retrieve-dataset.py 302b18f4-92d7-49e5-b7ea-0240cae81291 --output dataset.bin
Full supplier documentation
This dataset contains paragraph-level embeddings of the June 2024 Wikipedia dump for the 11 most popular languages (en, de, es, fa, fr, it, ja, pt, ru, tr, zh), generated using the multilingual BGE-M3 model. Articles are split into paragraphs, prefixed with their article titles to improve search quality, and short paragraphs (<100 chars) are filtered out. The dataset is provided in Parquet format and is well-suited for: - Multilingual Retrieval-Augmented Generation (RAG) pipelines - Semantic search benchmarks and evals - Vector database population (Upstash Vector, Pinecone, Qdrant, Weaviate, etc.) - Cross-lingual retrieval research Licensed under Apache 2.0. Available via DataBazaar's unified MCP/REST interface for streaming and query access at scale. _Federated dataset: 76 parquet shards, 49.48 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: Wikipedia 2024-06 Embeddings (BGE-M3, Multilingual) Wikipedia paragraph embeddings from the June 2024 dump across 11 languages, generated with the multilingual BGE-M3 model. Ideal for multilingual RAG, semantic search, and retrieval evals.

Schema

NameTypeDescription
idVARCHARUnique identifier combining article ID and paragraph number (format: {article_id}_{paragraph_index})
urlVARCHARFull Wikipedia URL for the source article
titleVARCHARWikipedia article title in original language
textVARCHARParagraph text extracted from article body, prefixed with article title
embeddingFLOAT[]1024-dimensional dense vector from BGE-M3 model (multilingual)

Sample Data

Preview a sample of the data before downloading.

Public sample only. Sign in to retrieve the full dataset, including free datasets.

For AI Agents

Via MCP Server
# 1. Add to your agent's MCP config (claude_desktop_config.json or similar):
{
  "mcpServers": {
    "databazaar": { "command": "npx", "args": ["databazaar-mcp"] }
  }
}

# 2. Your agent can then call:
search_datasets({ query: "Wikipedia Embeddings BGE-M3 Mu" })
// Found: 302b18f4-92d7-49e5-b7ea-0240cae81291
get_download_url({ dataset_id: "302b18f4-92d7-49e5-b7ea-0240cae81291" })  // free — sign in with MCP OAuth first
Via REST API
# Free dataset — sign in or use your account API key:
curl https://api.databazaar.io/datasets/302b18f4-92d7-49e5-b7ea-0240cae81291/download-url -H "Authorization: Bearer $DATABAZAAR_API_KEY"