Method
“halo-halo tags Tagalog-English code-switching at the token and intra-word level — including affixes like nag-/i-/pinag-...-an attached to English roots — using a six-label scheme adapted from the CALCS code-switching tagset (Solorio et al., 2014) and Universal Dependencies' multi-word-token mechanism. It extends the granularity gap left by TweetTaglish (LREC 2022, tweet-level mixing proportion) and Batayan (ACL 2025, sentence-level Taglish sentiment/toxicity only) rather than reproducing either. The v1 segmenter is a transparent rule/lexicon baseline — every label traces to a published rule you can read. The eval set is a single-annotator v0 (James, native bilingual), calibrated by self test-retest agreement, not inter-annotator agreement — it supports a documented method and a demo calibration, not a population or state-of-the-art claim, and we are seeking co-annotators to change that.”
The exact claims-boundary sentence from this project's Linguistic Spec (docs/batch2-linguistic-spec.md §8.4) — copywriting owns tone, this spec owns what may truthfully be asserted.
Two-tier tokenization
The unit of analysis is not “word.” It is a two-tier token, defined by a three-stage pipeline:
- Tier 0 — social pre-extraction. URLs, @mentions, #hashtags, and emoji sequences (grapheme-cluster-correct, never a raw code-point count) are pulled out as atomic OTHER spans BEFORE any word-boundary rule runs — otherwise a URL fragments unpredictably on “.”/“/”.
- Tier 1 — UAX #29 word boundaries. Standard Unicode word segmentation over what Tier 0 left behind. Alone, this is structurally blind to intra-word switching — it sees “nag-book” as one opaque word.
- Tier 2 — the affix layer. Inside a Tier-1 span, a closed, versioned Tagalog affix lexicon splits grammatical prefixes/circumfixes from a foreign root — but ONLY when the root does not resolve to a recognized native Tagalog root first (the guard that stops a real Tagalog word like “pinagbutihan” from being torn into three meaningless pieces). This tier is the project.
The six-tag set
TAG / ENG / MIXED / NE / OTHER / AMBIGUOUS — adapted from the CALCS shared-task tagset (Solorio et al. 2014). AMBIGUOUS is first-class, not a residual bucket to minimize.
| Tag | Decision rule |
|---|---|
| TAG | Carries Tagalog grammar or lexical meaning, after respelling normalization. |
| ENG | An attested English lexeme not (yet) productively taking native Tagalog inflection. |
| MIXED | One token, morphemes from both languages, that Tier 2's edge-matching cannot cleanly separate (non-concatenative morphology — infixation). |
| NE | Proper nouns opt out of the TAG/ENG axis by referent type, not etymology or capitalization. |
| OTHER | Punctuation, emoji, URLs, bare numerals, paralinguistic markers — no language-specific content. |
| AMBIGUOUS | Cannot be resolved with confidence in good faith. Always paired with a confidence flag and, where possible, a reason. Never a silent default. |
Ten worked examples
The full 20-row table lives in docs/batch2-linguistic-spec.md §2.3 and is reproduced as executable fixtures in src/core/segmenter.test.ts. The ten hardest are worth reading here directly:
| Input | Ruling | Why |
|---|---|---|
| nag-book | nag- = TAG (prefix), book = ENG (root) | Tier-2 prefix match at the left edge; the root is an unnativized English lexeme. |
| i-explain / iexplain | i- = TAG, explain = ENG (identical with or without the hyphen) | The hyphen is a confidence booster, not a requirement. |
| pinag-research-an | pinag- = TAG, research = ENG, -an = TAG | A circumfix wraps a foreign root on both sides — matched as ONE two-attachment-point entry, never two independent strips. |
| jan (lowercase) | TAG, LOW confidence | A respelling of "diyan" — but ambiguous with the given name "Jan". |
| Jan (capitalized) | NE | Same string, different capitalization, different referent — the demo's canonical capitalization-sensitive case. |
| ATM | ENG, not NE | Refers to a class of machine, not a specific named entity — the NE/ENG line is drawn by referent type, never by capitalization or brand-like shape. |
| gets | AMBIGUOUS, LOW confidence | Visibly English in spelling, but takes native Tagalog affixation ("nagets") for many speakers — a label encodes a labeling regime, not a fact. |
| dinownload | MIXED, one token | The Tagalog perfective infix -in- lands INSIDE the English root, not at an edge — Tier 2's edge-matching lexicon structurally cannot bisect this. The flagship hard case. |
| magandang bahay | maganda = TAG, -ng = TAG (linker, MWT-split), bahay = TAG | Pure-Tagalog sanity check — the MWT mechanism is a general Tagalog phenomenon (Universal Dependencies already handles it), not invented for code-switching. |
| hahaha | OTHER | A paralinguistic marker attested identically in English- and Tagalog-language writing — no fact of the matter about its language. |
Switch points
A switch point is a boundary between two adjacent tokens whose labels differ, counted ONLY over the {TAG, ENG} pair. A boundary touching OTHER, NE, AMBIGUOUS, or MIXED is not counted — that would conflate referent-type and discourse-marker boundaries with actual bilingual grammatical switching. A Tier-2-internal boundary (between “nag-” and “book”) IS a switch point — showing that intra-word switch points exist at all, and where, is this project's actual reason to exist.
Every label traces to a rule
The live demo's rule-trace panel shows, per token, exactly which of the following 24 registered rules fired — the same registry ships in src/core/rules.ts, openable by anyone.
Claims ceiling
| CAN honestly support | CANNOT honestly support |
|---|---|
| A documented, versioned method for tagging Taglish at token and intra-word granularity, with worked decision rules a second annotator could pick up. | Any population-level claim (“how Filipinos code-switch,” “X% of Taglish tokens are…”) — one author's fixtures are not a sample of a population. |
| “The demo segmenter was calibrated against N fixture items” — a narrowly-scoped demo-calibration claim. | Any SOTA or “outperforms” comparison against TweetTaglish, Batayan, LinCE, or the CalamanCy-adjacent papers — none of them attempt this exact task, so there is no shared benchmark, and n=1 with no cross-annotator agreement cannot support a comparative accuracy claim regardless. |
| Descriptive claims sourced to the linguistics literature (Bautista 2004's switch-type proportions) as independently published findings. | Any claim that this label set or its rulings are correct in a normative sense — a label encodes one bilingual annotator's regime, not the truth. |
Lineage credits
- CALCS shared-task tagset — Solorio et al. 2014 (aclanthology.org/W14-3907/) — the six-tag scheme's origin.
- Universal Dependencies — foreign-expression guidelines and the Multi-Word-Token mechanism (universaldependencies.org) — the MWT split's precedent.
- TweetTaglish — Herrera, Aich, Parde, LREC 2022 — the tweet-level task-granularity gap this project extends.
- Batayan — ACL 2025 — the sentence-level task-granularity gap this project extends.
- LinCE — LREC 2020 — confirms Tagalog-English is not one of its four covered pairs.
- ark-tweet-nlp / Twokenize — the Tier-0 pre-tokenization engineering pattern (cited for the pattern only, not language coverage).
- calamanCy — Miranda, NLP-OSS 2023 — the base Tagalog NLP toolkit this project's Tier 2 does not touch.
- “Code-Switching Detection and Processing in Filipino-English Text Using CalamanCy” (ResearchGate 401083773, venue/authorship UNVERIFIED) — the nearest prior-art mechanism, a fine-tuned mBERT classifier, cited per /limitations's full disclosure.
- Bautista (2004) — the sociolinguistic grounding for the switch-point construct.
Single-annotator honesty
“v0, single bilingual annotator (James, native), self-test-retest reliability only — seeking co-annotators” ships in the same breath as every kappa or accuracy number this project publishes. The full annotation guideline, worked examples, and test-retest protocol are committed at docs/annotation-guideline-v1.md — dated and versioned BEFORE any item in the eval set was labeled. See /eval for the current numbers and /annotate for the tool that produces the real human pass.