API reference¶

Every public name of esperantilo, with its signature, arguments and examples.

English only

This page is generated from the docstrings of the source code, which are written in English. It is not translated into the other languages of this site.

Generated from the source

Nothing on this page is written by hand. It is built from the docstrings of src/esperantilo/ by mkdocstrings every time the site is built, so it cannot contradict the code. The >>> examples are executed by pytest too — see Guide → Testing.

Every entry links to the exact lines it documents: expand Source to read them.

esperantilo ¶

Esperantilo: a rule-based Natural Language Processing library for Esperanto.

This module is the public API of the library: everything a user is meant to touch is importable straight from it, whichever subpackage it actually lives in.

import esperantilo

page = esperantilo.WikiPage.look_up("Esperanto", language="eo")
esperantilo.sentence_tokenizer(page.section(page.title))

Two subpackages, split by what they do to text:

Subpackage	What it does
`esperantilo.wiki`	Fetches text: Wikipedia articles, as plain text, in any language.
`esperantilo.nlp`	Analyses text that is already in hand.

The library is deliberately incomplete — it currently ships sentence segmentation and a Wikipedia reader, and nothing else. Full documentation is at https://jparisu.github.io/nlp-esperantilo/.

esperantilo.version `module-attribute` ¶

__version__ = '0.1.0'

The installed version of the library, as declared in pyproject.toml.

esperantilo.nlp ¶

Natural Language Processing over Esperanto text.

The rule-based half of the library: everything that takes text that is already in hand and analyses it. Fetching the text in the first place is the job of esperantilo.wiki.

from esperantilo import sentence_tokenizer

sentence_tokenizer("Mi lernas Esperanton. Ĝi estas facila.")

One feature so far, sentence segmentation. Tokens, lemmas and affix analysis are still a design target — building them is the exercise the guide prepares you for.

sentence_tokenizer ¶

sentence_tokenizer(text: str, end_of_sentence: Iterable[str] = ('.', '!', '?')) -> list[str]

Split text into sentences.

A sentence ends at one of the end_of_sentence characters, which stays with the sentence it closes.

Parameters:

Name	Type	Description	Default
`text`	`str`	The text to split.	required
`end_of_sentence`	`Iterable[str]`	The characters that end a sentence. Defaults to `(".", "!", "?")`.	`('.', '!', '?')`

Returns:

Type	Description
`list[str]`	The sentences found, each stripped of surrounding whitespace. Empty ones
`list[str]`	are dropped, so a text of only whitespace yields an empty list.

Examples:

>>> sentence_tokenizer("Mi lernas Esperanton. Ĉu vere? Jes!")
['Mi lernas Esperanton.', 'Ĉu vere?', 'Jes!']

ToDo

Do not split on a dot inside a token, like 3.14 or esperanto.net.
Do not split after a known abbreviation, like Dr. or Mr..

Source code in src/esperantilo/nlp/tokenizer.py

def sentence_tokenizer(
    text: str,
    end_of_sentence: Iterable[str] = (".", "!", "?"),
) -> list[str]:
    """Split `text` into sentences.

    A sentence ends at one of the `end_of_sentence` characters, which stays with
    the sentence it closes.

    Args:
        text: The text to split.
        end_of_sentence: The characters that end a sentence. Defaults to
            `(".", "!", "?")`.

    Returns:
        The sentences found, each stripped of surrounding whitespace. Empty ones
        are dropped, so a text of only whitespace yields an empty list.

    Examples:
        >>> sentence_tokenizer("Mi lernas Esperanton. Ĉu vere? Jes!")
        ['Mi lernas Esperanton.', 'Ĉu vere?', 'Jes!']

    ToDo:
        - Do not split on a dot inside a token, like `3.14` or `esperanto.net`.
        - Do not split after a known abbreviation, like `Dr.` or `Mr.`.
    """
    sentences = [text]

    # Cut the text after every end-of-sentence character.
    for end_char in end_of_sentence:
        new_sentences = []

        # For each sentence in the input
        for sentence in sentences:

            # Separate by char
            sentence_separation = sentence.split(end_char)

            # Strip whitespace
            sentence_separation = [s.strip() for s in sentence_separation]

            # Add the end-of-sentence character back to all but the last piece.
            # The last piece is whatever followed the final character
            sentence_separation = [
                s + end_char for s in sentence_separation[:-1]
            ] + sentence_separation[-1:]

            # Remove empty sentences
            sentence_separation = [s for s in sentence_separation if s]

            # Store the sentences found
            new_sentences.extend(sentence_separation)

        # Convert the list of sentences found into the input for the next iteration
        sentences = new_sentences

    return sentences

esperantilo.wiki ¶

Reading Wikipedia articles as plain text, in any language.

Where the text comes from, as opposed to what is done with it afterwards — that half is esperantilo.nlp.

The whole feature is one class, WikiPage, and it knows how to fetch itself:

from esperantilo import WikiPage

page = WikiPage.look_up("Perejil", language="eo")
print(page.title)               # Petroselo
print(page.section("Uzoj"))

How the article is found — which APIs are called, in which order, and what happens when a language does not have it — is the business of the private esperantilo.wiki._wiki_api module, not of anyone using this one.

WikiPage `dataclass` ¶

WikiPage(title: str, language: str, qid: str, url: str, sections: Dict[str, str] = dict())

One Wikipedia article, in one language, as plain text.

Build one with look_up. The constructor is plain enough to call directly in tests.

The body is split into sections, keyed by heading, in the order the article presents them. Only top-level headings (== … ==) open a new section; deeper ones stay inside the text of the section they belong to, so nothing is lost and full_text() gives back the article as it was fetched.

Wikipedia leaves the lead — the text before the first heading — untitled, so it is keyed by the title of the article itself.

Parameters:

Name	Type	Description	Default
`title`	`str`	The title of the article, after any redirect was followed.	required
`language`	`str`	The language code of the wiki it came from, such as `"eo"`.	required
`qid`	`str`	The Wikidata identifier of the concept, such as `"Q143"`.	required
`url`	`str`	The canonical address of the article.	required
`sections`	`Dict[str, str]`	The body, mapping each heading to its text, with the lead under the title of the article.	`dict()`

Examples:

>>> page = WikiPage(
...     title="Esperanto",
...     language="eo",
...     qid="Q143",
...     url="https://eo.wikipedia.org/wiki/Esperanto",
...     sections={
...         "Esperanto": "Esperanto estas planlingvo.",
...         "Historio": "Zamenhof publikigis ĝin en 1887.",
...     },
... )
>>> page.section_names
['Esperanto', 'Historio']
>>> page.section("historio")
'Zamenhof publikigis ĝin en 1887.'

section_names `property` ¶

section_names: List[str]

The headings of the article, in the order they appear.

look_up `classmethod` ¶

look_up(title: str, language: str = 'en', fallback_languages: Sequence[str] = ('es', 'en', 'eo'), any_language: bool = True, timeout: Optional[float] = 10.0) -> WikiPage

Fetch the article a title refers to.

The title is looked for in language first and then in each of fallback_languages, stopping at the first wiki that knows it; redirects are followed. The article that comes back is the one for the concept found, in language if that wiki has it and in the first fallback that does otherwise — which is what makes searching in one language and reading in another a single call.

If no listed language knows the title and any_language is set, the concept is searched for across every language. That step is a search, not a lookup: it finds something for almost any plausible word, and what it finds is not necessarily what was asked for. Pass any_language=False to fail instead of guessing.

Parameters:

Name	Type	Description	Default
`title`	`str`	The title of the Wikipedia page.	required
`language`	`str`	The language to get the page in. Default is `"en"`.	`'en'`
`fallback_languages`	`Sequence[str]`	Languages to fall back on, both when searching for the title and when choosing which article to return.	`('es', 'en', 'eo')`
`any_language`	`bool`	If True, search across all languages when no listed language knows the title.	`True`
`timeout`	`Optional[float]`	Seconds to wait for each request, or None to wait forever. Default is 10.0 seconds.	`10.0`

Returns:

Type	Description
`WikiPage`	The article, in the first available language of the preference list.

Raises:

Type	Description
`ValueError`	If the title is not found in any of the specified languages, or if the concept has no article in any of them.
`RequestException`	If a request fails, times out, or the API answers with an error status.

Example

page = WikiPage.look_up("Perejil", language="eo")
print(page.title)   # Petroselo

Source code in src/esperantilo/wiki/wiki.py

@classmethod
def look_up(
        cls,
        title: str,
        language: str = "en",
        fallback_languages: Sequence[str] = ("es", "en", "eo"),
        any_language: bool = True,
        timeout: Optional[float] = 10.0,
) -> WikiPage:
    """Fetch the article a title refers to.

    The title is looked for in `language` first and then in each of
    `fallback_languages`, stopping at the first wiki that knows it;
    redirects are followed. The article that comes back is the one for the
    *concept* found, in `language` if that wiki has it and in the first
    fallback that does otherwise — which is what makes searching in one
    language and reading in another a single call.

    If no listed language knows the title and `any_language` is set, the
    concept is searched for across every language. That step is a *search*,
    not a lookup: it finds something for almost any plausible word, and what
    it finds is not necessarily what was asked for. Pass
    `any_language=False` to fail instead of guessing.

    Args:
        title: The title of the Wikipedia page.
        language: The language to get the page in. Default is `"en"`.
        fallback_languages: Languages to fall back on, both when searching
            for the title and when choosing which article to return.
        any_language: If True, search across all languages when no listed
            language knows the title.
        timeout: Seconds to wait for each request, or None to wait forever.
            Default is 10.0 seconds.

    Returns:
        The article, in the first available language of the preference list.

    Raises:
        ValueError: If the title is not found in any of the specified
            languages, or if the concept has no article in any of them.
        requests.RequestException: If a request fails, times out, or the
            API answers with an error status.

    Example:
        ```python
        page = WikiPage.look_up("Perejil", language="eo")
        print(page.title)   # Petroselo
        ```
    """
    return cls(*_wiki_api._fetch_by_title(
        title, language, fallback_languages, any_language, timeout
    ))

section ¶

section(name: str) -> str

Return the text of one section.

The name is matched ignoring case and surrounding whitespace: a heading is prose written by an editor, and its exact capitalisation is not something the caller should have to know.

Parameters:

Name	Type	Description	Default
`name`	`str`	The heading to look for.	required

Returns:

Type	Description
`str`	The text of that section, without its heading.

Raises:

Type	Description
`KeyError`	If the article has no such section.

Source code in src/esperantilo/wiki/wiki.py

def section(self, name: str) -> str:
    """Return the text of one section.

    The name is matched ignoring case and surrounding whitespace: a heading
    is prose written by an editor, and its exact capitalisation is not
    something the caller should have to know.

    Args:
        name: The heading to look for.

    Returns:
        The text of that section, without its heading.

    Raises:
        KeyError: If the article has no such section.
    """
    wanted = name.strip().casefold()
    for key, body in self.sections.items():
        if key.strip().casefold() == wanted:
            return body
    raise KeyError(f"No section {name!r}. Available: {self.section_names}")

full_text ¶

full_text() -> str

Return the whole article as one string, headings included.

Examples:

>>> page = WikiPage("Esperanto", "eo", "Q143", "", {"Historio": "En 1887."})
>>> print(page.full_text())
== Historio ==
En 1887.

Source code in src/esperantilo/wiki/wiki.py

def full_text(self) -> str:
    """Return the whole article as one string, headings included.

    Examples:
        >>> page = WikiPage("Esperanto", "eo", "Q143", "", {"Historio": "En 1887."})
        >>> print(page.full_text())
        == Historio ==
        En 1887.
    """
    return "\n\n".join(
        f"== {name} ==\n{body}" for name, body in self.sections.items()
    )

len ¶

__len__() -> int

The number of sections — not the length of the text.

Examples:

>>> len(WikiPage("Esperanto", "eo", "Q143", "", {"Historio": "En 1887."}))
1

Source code in src/esperantilo/wiki/wiki.py

def __len__(self) -> int:
    """The number of sections — *not* the length of the text.

    Examples:
        >>> len(WikiPage("Esperanto", "eo", "Q143", "", {"Historio": "En 1887."}))
        1
    """
    return len(self.sections)

getitem ¶

__getitem__(key: str) -> str

Read a section, as page["Historio"].

Delegates to section, so both ways of reading a section match names the same way.

Source code in src/esperantilo/wiki/wiki.py

def __getitem__(self, key: str) -> str:
    """Read a section, as `page["Historio"]`.

    Delegates to [`section`][esperantilo.wiki.WikiPage.section], so both
    ways of reading a section match names the same way.
    """
    return self.section(key)

contains ¶

__contains__(key: str) -> bool

Test for a section, as "Historio" in page, ignoring case.

Examples:

>>> "HISTORIO" in WikiPage("Esperanto", "eo", "Q143", "", {"Historio": "."})
True

Source code in src/esperantilo/wiki/wiki.py

def __contains__(self, key: str) -> bool:
    """Test for a section, as `"Historio" in page`, ignoring case.

    Examples:
        >>> "HISTORIO" in WikiPage("Esperanto", "eo", "Q143", "", {"Historio": "."})
        True
    """
    try:
        self.section(key)
    except KeyError:
        return False
    return True

iter ¶

__iter__() -> Iterator[str]

Iterate over the section names, as a mapping would.

Source code in src/esperantilo/wiki/wiki.py

def __iter__(self) -> Iterator[str]:
    """Iterate over the section names, as a mapping would."""
    return iter(self.sections)

str ¶

__str__() -> str

The article as plain text — the same as full_text().

Source code in src/esperantilo/wiki/wiki.py

def __str__(self) -> str:
    """The article as plain text — the same as `full_text()`."""
    return self.full_text()

repr ¶

__repr__() -> str

A one-line summary.

The repr a dataclass generates would dump every section, which is the whole article: unusable in a REPL or a notebook, which is exactly where a page is most often looked at.

Examples:

>>> WikiPage("Esperanto", "eo", "Q143", "", {"Historio": "En 1887."})
WikiPage(title='Esperanto', language='eo', qid='Q143', sections=1)

Source code in src/esperantilo/wiki/wiki.py

def __repr__(self) -> str:
    """A one-line summary.

    The repr a dataclass generates would dump every section, which is the
    whole article: unusable in a REPL or a notebook, which is exactly where
    a page is most often looked at.

    Examples:
        >>> WikiPage("Esperanto", "eo", "Q143", "", {"Historio": "En 1887."})
        WikiPage(title='Esperanto', language='eo', qid='Q143', sections=1)
    """
    return (
        f"WikiPage(title={self.title!r}, language={self.language!r}, "
        f"qid={self.qid!r}, sections={len(self.sections)})"
    )

API reference¶

esperantilo ¶

esperantilo.__version__ module-attribute ¶

esperantilo.nlp ¶

sentence_tokenizer ¶

esperantilo.wiki ¶

WikiPage dataclass ¶

section_names property ¶

look_up classmethod ¶

section ¶

full_text ¶

__len__ ¶

__getitem__ ¶

__contains__ ¶

__iter__ ¶

__str__ ¶

__repr__ ¶

esperantilo.version `module-attribute` ¶

WikiPage `dataclass` ¶

section_names `property` ¶

look_up `classmethod` ¶

len ¶

getitem ¶

contains ¶

iter ¶

str ¶

repr ¶