A large share of the input our systems receive is not in one language. It is in two, frequently in a script belonging to neither of them by convention, and often with a spelling that exists nowhere but in the head of the person typing.
Redrob serves twelve Indian languages and accepts code-mixed input on every surface. That second commitment is the expensive one, and it is not optional: the alternative is a product that asks a user to decide which language they are about to think in before they start typing.
This note is about where that input breaks a conventional pipeline, in the order it breaks.
The shape of the problem
A representative query, lightly edited:
mala pune madhe java developer chi job pahije, 6 lakh package
Marathi grammar. English technical nouns. Latin script throughout. An Indian numbering unit. No diacritics, no standard transliteration scheme, and a word order that neither an English nor a Marathi parser expects.
This is not an edge case that appears in a long tail. For several of the languages we serve it is the majority form of typed input, and it becomes more common the further you get from metropolitan users, which is exactly the population most likely to be underserved already.
Failure one: language identification
Standard language identification runs over the whole string and returns a single label with a confidence score. On the query above it returns English with reasonable confidence, because Latin script and the tokens java, developer, job and package dominate the character distribution.
Everything downstream then treats a Marathi sentence as an English one.
The fix is not a better classifier. It is a different output type. Language identification has to run per span and return a sequence, not per document returning a scalar. Once the tagger can say that tokens one through three are Marathi and tokens four through five are English, the rest of the pipeline has something to work with.
Failure two: normalisation destroys information
The instinct with messy input is to normalise it into a canonical form. Transliterate everything into the native script, standardise spellings, then parse.
This loses two things that matter.
Transliterating java developer into Devanagari produces a string that no job posting contains, so retrieval degrades. Technical vocabulary in Indian-language input is overwhelmingly written in Latin script, and forcing it into the native script moves it away from the corpus rather than towards it.
The second loss is subtler. The script a user chooses carries information about how they want to be answered. Someone typing Marathi in Latin script is usually most comfortable reading it that way. Normalising the input erases that preference before anything can act on it.
We keep the original span and its script tag alongside the normalised form, and let the retrieval and generation stages choose which they want.
Failure three: entity resolution across scripts
pune, Pune, पुणे and poona are one city. nagar is a suffix on dozens of place names and a place name on its own. Someone writing blr means Bengaluru and someone writing banglore means the same thing with a spelling that will never appear in a gazetteer.
Fuzzy matching on edit distance handles the misspellings and fails on the transliterations, because the edit distance between पुणे and poona is meaningless. Matching on a phonetic key handles the transliterations and produces false positives across genuinely different names.
What worked was resolving against a canonical entity set with both an orthographic and a phonetic index, then disambiguating with the rest of the query rather than in isolation. nagar next to a job title behaves differently from nagar next to a state name.
Failure four: the numbering system
6 lakh is six hundred thousand. So is 6L, 6 lac, 600000, 6,00,000 and, in some postings, 6 LPA where the annual part is implied by context rather than stated.
An English number parser reads none of these correctly except the bare digits, and 6,00,000 is actively dangerous because a parser expecting three-digit grouping reads it as six hundred thousand only by accident, or as six followed by two groups it cannot interpret.
This is a small problem with an outsized effect, because the number is usually the constraint the user cares most about. Getting the city wrong returns bad results. Getting the salary wrong returns results that insult the user.
What we changed
The pipeline now runs span-level language identification before anything else, preserves the original alongside the normalised form, resolves entities against a bilingual index, and parses quantities with a locale-aware grammar rather than a numeric one.
The largest gains came from the first and last of those, which is not what we expected going in. We had assumed the hard part would be semantic. It was mostly clerical.
What is still open
Two things we have not solved.
Span-level tagging is unreliable at boundaries when a single word is borrowed rather than switched, and the distinction between borrowing and switching is not always well defined even to a native speaker.
Evaluation remains the binding constraint. For most of these languages there is no public benchmark for code-mixed retrieval, which means every improvement we make is measured against an instrument we built ourselves. That is a problem we write about separately.
