Elijah Rippeth

Elijah Rippeth

Bad at computering

MTMA22 Day 1: Project Definition

3 minute read

Published:

This year I’m fortunate to be co-leading a Machine Translation Marathon in the Americas project with Marcin Junczys-Dowmunt. The project has modest goals: to add Python bindings to Marian NMT.

Marian, being a C++ toolkit, enjoys speed afforded by close-to-metal programming with fine-grained memory control. However, being a C++ library introduces a lot of complexity for those unfamiliar with C++, including dependency management, debugging, installation, and embedding into other applications. A common request for Marian developers is to add Python bindings to help mitigate some of these issues. We set out on this MTMA project with the goal is adding shallow bindings for some of the most common classes that we expect to be used by Python developers, starting with inference.

There are many ways to add Python bindings to existing C++ code ranging from Cython to SWIG to CFFI. We elected to use Pybind11 following the philosophy that, since Marian is a C++ library, adding Python bindings using C++ makes sense to reduce maintenance burden.

We started by defining a high-level Python API and example usage we expect users to want to follow and landed on the following mocked up API:

import pymarian
import sentencepiece as spm

src_spm = spm.SentencePieceProcessor(source_spm_model)
tgt_spm = spm.SentencePieceProcessor(target_spm_model)

# may change
translator = pymarian.Translator(yaml_file)

for sent in sents_to_translate:
    src_pieces_str = " ".join(src_spm.Encode(sent))
    tgt_pieces = translator.translate(src_pieces_str).split()
    translation = " ".join(tgt_spm.Decode(tgt_pieces))
    print(f"Translation of '{sent}' is '{translation}'")

We first needed to identify the right bit of Marian to wrap to achieve this API. We landed on marian::TranslateService<Search> as the most appropriate class to wrap and identified a couple of snags.

  1. marian::TranslateService<Search>::run only handles string-in, string-out translation which batches internally. This is fine as is, but we may expect users to want to pass lists of strings without requiring unnecessary joins and splits. This is a TODO.
  2. marian::TranslateService<Search> is constructed with using a pointer to a marian::Options object, which is similar to a lookup table which can be parsed from a number of configuration formats. To iterate quickly we locally added a C++ constructor which loads a YAML file directly. We may want to clean this up to be a factory instead, but that’s a TODO.

Now we have C++ and Python API parity for inference. We begun digging into the Pybind11 bit which was extremely straightforward in the initial pass:

#include "pybind11/pybind11.h"

#include "marian.h"
#include "translator/translator.h"
#include "translator/beam_search.h"

namespace py = pybind11;

PYBIND11_MODULE(pymarian, m) {
  // Classes
  py::class_<marian::TranslateService<marian::BeamSearch>>(m, "Translator")
      .def(py::init<std::string>())
      .def("translate", &marian::TranslateService<marian::BeamSearch>::run);
}

This just defines a Python module pymarian containing a single class Translator with a str-constructor (corresponding to the YAML file) and a str-in, str-out translate method which just calls te run method described above.

In total this took about 30 minutes. We spent much of the rest of the day fiddling with setuptools and cmake to make (heh) sure they can speak to one another. The details are a bit gritty so I won’t describe them here but after a single day we nearly have a working Python implementation of Marian inference. I expect on day 2 we will harden this API, add more functionality, and begin benchmarking the code to see the delta between native and wrapped code.

You May Also Enjoy

MTMA22 Day 5: Continuous Integration and Finalizing Packaging

less than 1 minute read

Published:

Day 5 is the final day of MTMA and we were prepping for our end-of-day demo. We largely had everything in place that we wanted, but are looking forwards to standardizing the build, packaging, and distribution of pymarian. I spent a lot of day fixing CI by making sure the right cmake flags were set and setting up python environments as appropriate. Meanwhile I learned that twine cannot upload arbitrary .whl files built for linux. We need to repair with auditwheel which is no big deal. Our CI matrix will eventually need to be something like python_versions X {cpu,gpu} X cuda_versions, but it will be worth it for consumers who no longer need to compile marian from source and can use it on all major platforms.

MTMA22 Day 4: Benchmarking and Applications

4 minute read

Published:

Now that the package is mostly solid, I spent the day benchmarking the code. I previously developed a small microbenchmark to test native vs. pybind speeds to ensure there are no major regressions, but I wanted to check against other existing libraries. Because we’re using Helsinki NLP’s OPUS MT models to test the Marian inference, we thought we would use the HuggingFace Transformer’s exported checkpoints of the same models. I’m not exactly sure how the checkpoints were made compatible (likely model surgery like how HF exports fairseq checkpoints), but it mostly doesn’t matter.

MTMA22 Day 3: Packaging

less than 1 minute read

Published:

With the code looking better and the build system improved, day 3 is primarily spent working to prepare pymarian for packaging and automated builds as a first class citizen in Marian CI. I was unavailable for most of the day due to internship obligations and some wedding planning, so I don’t have much to report. Marcin seems to have gotten pymarian building on Windows with some minor cmake magic. Meanwhile Alex checked in some of the initial Huggingface-like API’s and Liling has been fighting with CI. That’s really it for today. :-)

MTMA22 Day 2: Hardening the API

3 minute read

Published:

After yesterday’s productive day of fighting with setuptools and cmake we were able to get a kind-of-awkward-to-use pymarian translator working. Our outstanding TODO’s were to test the speed (to ensure we aren’t incurring terrible latency by adding bindings) and to clean up some of the API’s.