Added in API level 24

Normalizer2


abstract class Normalizer2
kotlin.Any
   ↳ android.icu.text.Normalizer2

Unicode normalization functionality for standard Unicode normalization or for using custom mapping tables. All instances of this class are unmodifiable/immutable. The Normalizer2 class is not intended for public subclassing.

The primary functions are to produce a normalized string and to detect whether a string is already normalized. The most commonly used normalization forms are those defined in https://www.unicode.org/reports/tr15/ However, this API supports additional normalization forms for specialized purposes. For example, NFKC_Casefold is provided via getInstance("nfkc_cf", COMPOSE) and can be used in implementations of UTS #46.

Not only are the standard compose and decompose modes supplied, but additional modes are provided as documented in the Mode enum.

Some of the functions in this class identify normalization boundaries. At a normalization boundary, the portions of the string before it and starting from it do not interact and can be handled independently.

The spanQuickCheckYes() stops at a normalization boundary. When the goal is a normalized string, then the text before the boundary can be copied, and the remainder can be processed with normalizeSecondAndAppend().

The hasBoundaryBefore(), hasBoundaryAfter() and isInert() functions test whether a character is guaranteed to be at a normalization boundary, regardless of context. This is used for moving from one normalization boundary to the next or preceding boundary, and for performing iterative normalization.

Iterative normalization is useful when only a small portion of a longer string needs to be processed. For example, in ICU, iterative normalization is used by the NormalizationTransliterator (to avoid replacing already-normalized text) and ucol_nextSortKeyPart() (to process only the substring for which sort key bytes are computed).

The set of normalization boundaries returned by these functions may not be complete: There may be more boundaries that could be returned. Different functions may return different boundaries.

Summary

Nested classes

Constants for normalization modes.

Public methods
abstract StringBuilder!
append(first: StringBuilder!, second: CharSequence!)

Appends the second string to the first string (merging them at the boundary) and returns the first string.

open Int
composePair(a: Int, b: Int)

Performs pairwise composition of a & b and returns the composite if there is one.

open Int

Gets the combining class of c.

abstract String!

Gets the decomposition mapping of c.

open static Normalizer2!
getInstance(data: InputStream!, name: String!, mode: Normalizer2.Mode!)

Returns a Normalizer2 instance which uses the specified data file (an ICU data file if data=null, or else custom binary data) and which composes or decomposes text according to the specified mode.

open static Normalizer2!

Returns a Normalizer2 instance for Unicode NFC normalization.

open static Normalizer2!

Returns a Normalizer2 instance for Unicode NFD normalization.

open static Normalizer2!

Returns a Normalizer2 instance for Unicode toNFKC_Casefold() normalization which is equivalent to applying the NFKC_Casefold mappings and then NFC.

open static Normalizer2!

Returns a Normalizer2 instance for Unicode NFKC normalization.

open static Normalizer2!

Returns a Normalizer2 instance for Unicode NFKD normalization.

open String!

Gets the raw decomposition mapping of c.

abstract Boolean

Tests if the character always has a normalization boundary after it, regardless of context.

abstract Boolean

Tests if the character always has a normalization boundary before it, regardless of context.

abstract Boolean

Tests if the character is normalization-inert.

abstract Boolean

Tests if the string is normalized.

open String!

Returns the normalized form of the source string.

abstract StringBuilder!

Writes the normalized form of the source string to the destination string (replacing its contents) and returns the destination string.

abstract Appendable!

Writes the normalized form of the source string to the destination Appendable and returns the destination Appendable.

abstract StringBuilder!

Appends the normalized form of the second string to the first string (merging them at the boundary) and returns the first string.

abstract Normalizer.QuickCheckResult!

Tests if the string is normalized.

abstract Int

Returns the end of the normalized substring of the input string.

Public methods

append

Added in API level 24
abstract fun append(
    first: StringBuilder!,
    second: CharSequence!
): StringBuilder!

Appends the second string to the first string (merging them at the boundary) and returns the first string. The result is normalized if both the strings were normalized. The first and second strings must be different objects.

Parameters
first StringBuilder!: string, should be normalized
second CharSequence!: string, should be normalized
Return
StringBuilder! first

composePair

Added in API level 24
open fun composePair(
    a: Int,
    b: Int
): Int

Performs pairwise composition of a & b and returns the composite if there is one.

Returns a composite code point c only if c has a two-way mapping to a+b. In standard Unicode normalization, this means that c has a canonical decomposition to a+b and c does not have the Full_Composition_Exclusion property.

This function is independent of the mode of the Normalizer2. The default implementation returns a negative value.

Parameters
a Int: A (normalization starter) code point.
b Int: Another code point.
Return
Int The non-negative composite code point if there is one; otherwise a negative value.

getCombiningClass

Added in API level 24
open fun getCombiningClass(c: Int): Int

Gets the combining class of c. The default implementation returns 0 but all standard implementations return the Unicode Canonical_Combining_Class value.

Parameters
c Int: code point
Return
Int c's combining class

getDecomposition

Added in API level 24
abstract fun getDecomposition(c: Int): String!

Gets the decomposition mapping of c. Roughly equivalent to normalizing the String form of c on a DECOMPOSE Normalizer2 instance, but much faster, and except that this function returns null if c does not have a decomposition mapping in this instance's data. This function is independent of the mode of the Normalizer2.

Parameters
c Int: code point
Return
String! c's decomposition mapping, if any; otherwise null

getInstance

Added in API level 24
Deprecated in API level 31
open static fun getInstance(
    data: InputStream!,
    name: String!,
    mode: Normalizer2.Mode!
): Normalizer2!

Deprecated: Don't use because the binary data format is not stable across API levels.

Returns a Normalizer2 instance which uses the specified data file (an ICU data file if data=null, or else custom binary data) and which composes or decomposes text according to the specified mode. Returns an unmodifiable singleton instance.

  • Use data=null for data files that are part of ICU's own data.
  • Use name="nfc" and COMPOSE/DECOMPOSE for Unicode standard NFC/NFD.
  • Use name="nfkc" and COMPOSE/DECOMPOSE for Unicode standard NFKC/NFKD.
  • Use name="nfkc_cf" and COMPOSE for Unicode standard NFKC_CF=NFKC_Casefold.
If data!=null, then the binary data is read once and cached using the provided name as the key. If you know or expect the data to be cached already, you can use data!=null for non-ICU data as well.

Any java.io.IOException is wrapped into a android.icu.util.ICUUncheckedIOException.

Parameters
data InputStream!: the binary, big-endian normalization (.nrm file) data, or null for ICU data
name String!: "nfc" or "nfkc" or "nfkc_cf" or "nfkc_scf" or name of custom data file
mode Normalizer2.Mode!: normalization mode (compose or decompose etc.)
Return
Normalizer2! the requested Normalizer2, if successful

getNFCInstance

Added in API level 24
open static fun getNFCInstance(): Normalizer2!

Returns a Normalizer2 instance for Unicode NFC normalization. Same as getInstance(null, "nfc", Mode.COMPOSE). Returns an unmodifiable singleton instance.

Return
Normalizer2! the requested Normalizer2, if successful

getNFDInstance

Added in API level 24
open static fun getNFDInstance(): Normalizer2!

Returns a Normalizer2 instance for Unicode NFD normalization. Same as getInstance(null, "nfc", Mode.DECOMPOSE). Returns an unmodifiable singleton instance.

Return
Normalizer2! the requested Normalizer2, if successful

getNFKCCasefoldInstance

Added in API level 24
open static fun getNFKCCasefoldInstance(): Normalizer2!

Returns a Normalizer2 instance for Unicode toNFKC_Casefold() normalization which is equivalent to applying the NFKC_Casefold mappings and then NFC. See https://www.unicode.org/reports/tr44/#NFKC_Casefold

Same as getInstance(null, "nfkc_cf", Mode.COMPOSE). Returns an unmodifiable singleton instance.

Return
Normalizer2! the requested Normalizer2, if successful

getNFKCInstance

Added in API level 24
open static fun getNFKCInstance(): Normalizer2!

Returns a Normalizer2 instance for Unicode NFKC normalization. Same as getInstance(null, "nfkc", Mode.COMPOSE). Returns an unmodifiable singleton instance.

Return
Normalizer2! the requested Normalizer2, if successful

getNFKDInstance

Added in API level 24
open static fun getNFKDInstance(): Normalizer2!

Returns a Normalizer2 instance for Unicode NFKD normalization. Same as getInstance(null, "nfkc", Mode.DECOMPOSE). Returns an unmodifiable singleton instance.

Return
Normalizer2! the requested Normalizer2, if successful

getRawDecomposition

Added in API level 24
open fun getRawDecomposition(c: Int): String!

Gets the raw decomposition mapping of c.

This is similar to the getDecomposition() method but returns the raw decomposition mapping as specified in UnicodeData.txt or (for custom data) in the mapping files processed by the gennorm2 tool. By contrast, getDecomposition() returns the processed, recursively-decomposed version of this mapping.

When used on a standard NFKC Normalizer2 instance, getRawDecomposition() returns the Unicode Decomposition_Mapping (dm) property.

When used on a standard NFC Normalizer2 instance, it returns the Decomposition_Mapping only if the Decomposition_Type (dt) is Canonical (Can); in this case, the result contains either one or two code points (=1..4 Java chars).

This function is independent of the mode of the Normalizer2. The default implementation returns null.

Parameters
c Int: code point
Return
String! c's raw decomposition mapping, if any; otherwise null

hasBoundaryAfter

Added in API level 24
abstract fun hasBoundaryAfter(c: Int): Boolean

Tests if the character always has a normalization boundary after it, regardless of context. If true, then the character does not normalization-interact with following characters. In other words, a string containing this character can be normalized by processing portions up to this character and after this character independently. This is used for iterative normalization. See the class documentation for details.

Note that this operation may be significantly slower than hasBoundaryBefore().

Parameters
c Int: character to test
Return
Boolean true if c has a normalization boundary after it

hasBoundaryBefore

Added in API level 24
abstract fun hasBoundaryBefore(c: Int): Boolean

Tests if the character always has a normalization boundary before it, regardless of context. If true, then the character does not normalization-interact with preceding characters. In other words, a string containing this character can be normalized by processing portions before this character and starting from this character independently. This is used for iterative normalization. See the class documentation for details.

Parameters
c Int: character to test
Return
Boolean true if c has a normalization boundary before it

isInert

Added in API level 24
abstract fun isInert(c: Int): Boolean

Tests if the character is normalization-inert. If true, then the character does not change, nor normalization-interact with preceding or following characters. In other words, a string containing this character can be normalized by processing portions before this character and after this character independently. This is used for iterative normalization. See the class documentation for details.

Note that this operation may be significantly slower than hasBoundaryBefore().

Parameters
c Int: character to test
Return
Boolean true if c is normalization-inert

isNormalized

Added in API level 24
abstract fun isNormalized(s: CharSequence!): Boolean

Tests if the string is normalized. Internally, in cases where the quickCheck() method would return "maybe" (which is only possible for the two COMPOSE modes) this method resolves to "yes" or "no" to provide a definitive result, at the cost of doing more work in those cases.

Parameters
s CharSequence!: input string
Return
Boolean true if s is normalized

normalize

Added in API level 24
open fun normalize(src: CharSequence!): String!

Returns the normalized form of the source string.

Parameters
src CharSequence!: source string
Return
String! normalized src

normalize

Added in API level 24
abstract fun normalize(
    src: CharSequence!,
    dest: StringBuilder!
): StringBuilder!

Writes the normalized form of the source string to the destination string (replacing its contents) and returns the destination string. The source and destination strings must be different objects.

Parameters
src CharSequence!: source string
dest StringBuilder!: destination string; its contents is replaced with normalized src
Return
StringBuilder! dest

normalize

Added in API level 24
abstract fun normalize(
    src: CharSequence!,
    dest: Appendable!
): Appendable!

Writes the normalized form of the source string to the destination Appendable and returns the destination Appendable. The source and destination strings must be different objects.

Any java.io.IOException is wrapped into a android.icu.util.ICUUncheckedIOException.

Parameters
src CharSequence!: source string
dest Appendable!: destination Appendable; gets normalized src appended
Return
Appendable! dest

normalizeSecondAndAppend

Added in API level 24
abstract fun normalizeSecondAndAppend(
    first: StringBuilder!,
    second: CharSequence!
): StringBuilder!

Appends the normalized form of the second string to the first string (merging them at the boundary) and returns the first string. The result is normalized if the first string was normalized. The first and second strings must be different objects.

Parameters
first StringBuilder!: string, should be normalized
second CharSequence!: string, will be normalized
Return
StringBuilder! first

quickCheck

Added in API level 24
abstract fun quickCheck(s: CharSequence!): Normalizer.QuickCheckResult!

Tests if the string is normalized. For the two COMPOSE modes, the result could be "maybe" in cases that would take a little more work to resolve definitively. Use spanQuickCheckYes() and normalizeSecondAndAppend() for a faster combination of quick check + normalization, to avoid re-checking the "yes" prefix.

Parameters
s CharSequence!: input string
Return
Normalizer.QuickCheckResult! the quick check result

spanQuickCheckYes

Added in API level 24
abstract fun spanQuickCheckYes(s: CharSequence!): Int

Returns the end of the normalized substring of the input string. In other words, with end=spanQuickCheckYes(s); the substring s.subSequence(0, end) will pass the quick check with a "yes" result.

The returned end index is usually one or more characters before the "no" or "maybe" character: The end index is at a normalization boundary. (See the class documentation for more about normalization boundaries.)

When the goal is a normalized string and most input strings are expected to be normalized already, then call this method, and if it returns a prefix shorter than the input string, copy that prefix and use normalizeSecondAndAppend() for the remainder.

Parameters
s CharSequence!: input string
Return
Int "yes" span end index