textCohereLabs/wikipedia-2023-11-embed-multilingual-v3wikipediaembeddingsmultilingualragsemantic-searchcohereknowledge-basevector-search

Wikipedia Multilingual Embeddings

2023-11

Free

Open dataset

Sample structure: 100 / 100
1 download links issued
Seller: DataBazaar
Sign up to download

Already have an account? Log in

Agent? Connect your account →

Category
Text
Records
247,154,006 rows
Format
PARQUET
Update Frequency
Not documented
Collection Method
auto_imported_huggingface_federated
PII
No flagged field names; not a privacy audit
File Size
~511427.6 MB
Download links issued
1

Source, license and coverage

Supplier documentation. These claims are separate from the automated sample score. A listing edit date is not a data freshness date.

License
Not documented — confirm reuse terms with the seller
Source / creator
huggingface: CohereLabs/wikipedia-2023-11-embed-multilingual-v3
Collection method
auto_imported_huggingface_federated
Coverage / snapshot
2023-11
Data last updated
Not documented
Update schedule
Not documented

Source documentation ↗

Sample structure score: 100 / 100

This automated check describes the inspected sample, not factual accuracy, legal rights, representativeness, or the quality of the entire dataset. It is not a customer rating.

Assessed 10 sample records (JSON) on 2026-10-09. All records in the provided sample were checked.

CheckPointsEvidence
Populated cells50 / 5050 of 50 top-level cells contain a value. Null, absent and blank values count as missing; zero and false count as populated.
Consistent value types30 / 3050 of 50 populated cells match their column's most common observed type. Types are inferred, not checked against real-world truth.
Consistent record shape20 / 2010 of 10 records have the expected fields. CSV/TSV use the header width; JSON uses the union of observed keys.
Field-level findings and improvements

Check missing cells and mixed types below. Document intentional missing values or mixed types in your field descriptions. Do not fill legitimate unknowns with invented values just to increase this score.

FieldMissing cellsMost common typeOther populated types
_id0 / 10string0 / 10
url0 / 10string0 / 10
title0 / 10string0 / 10
text0 / 10string0 / 10
emb0 / 10object0 / 10
How the score is calculated, its limitations, and how to correct an assessment →

About this data

Semantic embeddings of Wikipedia paragraphs across 300+ languages, generated with Cohere Embed V3. Supports multilingual semantic search and retrieval-augmented generation tasks.

Retrieve with your agent or Python

Create an account and configure DATABAZAAR_API_KEY. This example retrieves free or already purchased data; it never makes a purchase. For a multi-file dataset, choose a file index from the manifest.

Download the Python example
python3 retrieve-dataset.py 5cb6f7d6-588b-4303-87c6-a33164c0d2c4 --output dataset.bin
Full supplier documentation
This dataset is a chunked-and-embedded version of the 2023-11-01 Wikipedia dump covering 300+ languages, produced by Cohere Labs. Each article is split into paragraphs and embedded with the state-of-the-art multilingual Cohere Embed V3 model, yielding close to 250 million vectors paired with their source text and metadata. It is one of the most widely used open knowledge bases for retrieval-augmented generation (RAG), multilingual semantic search, cross-lingual retrieval evaluation, and embedding-model benchmarking. The pre-computed embeddings save substantial compute compared to re-embedding Wikipedia from scratch. Via DataBazaar, agents can stream or query this dataset through a unified MCP/REST interface without downloading the full multi-terabyte corpus. Wikipedia content is licensed CC-BY-SA, and Cohere distributes the embeddings under terms compatible with redistribution. _Federated dataset: 2,707 parquet shards, 499.44 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: Wikipedia 2023-11 Multilingual Embeddings (Cohere Embed V3, 300+ Languages) ~250M Wikipedia paragraph embeddings across 300+ languages, generated with Cohere Embed V3. Ideal for multilingual semantic search and RAG.

Schema

NameTypeDescription
_idVARCHARUnique identifier combining Wikipedia language code, dump date, and chunk sequence (format: YYYYMMDD.ll_shard_chunk)
urlVARCHARFull URL to the source Wikipedia article
titleVARCHARArticle title in its original language
textVARCHARParagraph text chunk from the article
embFLOAT[]1024-dimensional embedding vector from Cohere Embed V3 model

Sample Data

Preview a sample of the data before downloading.

Public sample only. Sign in to retrieve the full dataset, including free datasets.

For AI Agents

Via MCP Server
# 1. Add to your agent's MCP config (claude_desktop_config.json or similar):
{
  "mcpServers": {
    "databazaar": { "command": "npx", "args": ["databazaar-mcp"] }
  }
}

# 2. Your agent can then call:
search_datasets({ query: "Wikipedia Multilingual Embeddi" })
// Found: 5cb6f7d6-588b-4303-87c6-a33164c0d2c4
get_download_url({ dataset_id: "5cb6f7d6-588b-4303-87c6-a33164c0d2c4" })  // free — sign in with MCP OAuth first
Via REST API
# Free dataset — sign in or use your account API key:
curl https://api.databazaar.io/datasets/5cb6f7d6-588b-4303-87c6-a33164c0d2c4/download-url -H "Authorization: Bearer $DATABAZAAR_API_KEY"