R/tokens-methods.R, R/tokens.R
as.tokens.RdCoercion functions to and from tokens objects, checks for whether an object is a tokens object, and functions to combine tokens objects.
# S3 method for tokens
as.list(x, ...)
# S3 method for tokens
as.character(x, use.names = FALSE, ...)
is.tokens(x)
as.tokens(x, concatenator = "_", ...)
# S3 method for spacyr_parsed
as.tokens(
x,
concatenator = "/",
include_pos = c("none", "pos", "tag"),
use_lemma = FALSE,
...
)
is.tokens(x)object to be coerced or checked
additional arguments used by specific methods. For c.tokens, these are the tokens objects to be concatenated.
logical; preserve names if TRUE. For
as.character and unlist only.
character between multi-word expressions, default is the underscore character. See Details.
character; whether and which part-of-speech tag to use:
"none" do not use any part of speech indicator, "pos" use the
pos variable, "tag" use the tag variable. The POS
will be added to the token after "concatenator".
logical; if TRUE, use the lemma rather than the raw
token
as.list returns a simple list of characters from a
tokens object.
as.character returns a character vector from a
tokens object.
is.tokens returns TRUE if the object is of class
tokens, FALSE otherwise.
as.tokens returns a quanteda tokens object.
is.tokens returns TRUE if the object is of class
tokens, FALSE otherwise.
The concatenator is used to automatically generate dictionary
values for multi-word expressions in tokens_lookup() and
dfm_lookup(). The underscore character is commonly used to join
elements of multi-word expressions (e.g. "piece_of_cake", "New_York"), but
other characters (e.g. whitespace " " or a hyphen "-") can also be used.
In those cases, users have to tell the system what is the concatenator in
your tokens so that the conversion knows to treat this character as the
inter-word delimiter, when reading in the elements that will become the
tokens.
# create tokens object from list of characters with custom concatenator
dict <- dictionary(list(country = "United States",
sea = c("Atlantic Ocean", "Pacific Ocean")))
lis <- list(c("The", "United-States", "has", "the", "Atlantic-Ocean",
"and", "the", "Pacific-Ocean", "."))
toks <- as.tokens(lis, concatenator = "-")
tokens_lookup(toks, dict)
#> Tokens consisting of 1 document.
#> text1 :
#> [1] "country" "sea" "sea"
#>