Job Batches: Ease Integration with the Smartling Jobs API
A Job is a container for content, such as files, strings, etc., to be translated across one or more languages. This container can be thought of as a unit of work, with a possible finite lifespan. Although Jobs are widely used in the Smartling Dashboard for packaging files, Smartling also exposes an API that makes it possible to integrate third-party content providers with the translation system. For example, Smartling provides connectors for systems such as Github, Contentful, Drupal, and many others. These connectors use a wide range of Smartling APIs, including Jobs, Files and Job Batches APIs. Let’s take a deeper look into how the new Job Batches API can be useful. However, we first need to look at the use case which is common amongst all the connectors and integrations.
Upload content flow
Every connector implements several flows, such as upload content for translation, check status of uploaded files, and download files with translations when they are done. These are basic flows, and most of the connectors implement some extra ones, such as attaching the context and synchronizing updated content with Smartling. For the sake of this article, we’ll specifically look at “upload flow”.
At a high level, the upload process looks like this:
- Select the necessary content (nodes, posts, pages, etc).
- Serialize the selected content entities into the XML or JSON, or whichever formats are supported by Smartling.
- Iterate through the files and upload them into to the TMS within the Job (a newly created or an existing one).
The first two steps are mostly related to the particular system you wish to integrate with, while the last step is the same for all integrations. Now let’s see how the third step can be implemented, and how the APIs are involved in the process.
Implementation
Files API and Jobs API
It’s possible to implement the upload flow directly by using the Files and Jobs API, but there are some difficulties related to this approach. The steps that you should take, in order to properly create a Job in Smartling, are as follows:
- Define the job (Jobs API)
1.1. Create or choose an existing job (sync operation).
2. Upload your original/source content (File API)
2.1. Upload the original files (async operation).
2.1.1. Wait until the files are fully ingested in the DB (polling).
3. Work with the job (Jobs API)
3.1. Add files to the job (async operation).
3.1.1. Wait until the files are added to the Job (polling).
3.2. Wrap-up job
3.2.1. Authorize job (sync operation)
In most cases, the first step expects interaction with the user, so the idea is that we start working as soon as we have a Job’s unique identifier.
For the second and third steps, this is where things can get complicated because there are some async operations that must be handled on the client side. Generally, adding multiple files to a Job is associated with polling (a file is parsed asynchronously and can’t be added to a Job until the parsing process is completed). Adding a file to a job is also asynchronous, and job can’t be authorized until all background tasks related to the Job are completed.
This forces third-party integration to poll every file after each step. For example, let’s take a look at the “Add files to job” operation. If the system can quickly add the file and its content to the Job, it synchronously responds with a 200 status. When the system cannot, a 202 response is returned which means you have to poll the process status on your own.
Having said this, it means that every single connector and integration must implement all of these on its own: polling and error handling logic. This can cause issues related to the maintenance of such code. You can imagine that if you have several connectors or integrations, or several platforms to support that are written in different languages, that might cause a huge issue to support the same logic in different places.
Job Batches API
The Job Batches API was written in order to handle all cases previously described, however, it also introduces the concept of a Job Batch.
Jobs Batches take care of all asynchronous processes while uploading and attaching files to Jobs. Batches hide this complexity from third-party integrations ie. after it has accepted the file from the client, it conducts this file through all steps, all the way up to authorizing the Job. Since the Jobs Batch Service uses not only the Jobs API, but also internal Smartling events, it becomes more effective than just polling files and jobs.
From the client side, upload flow consists of a few simple steps:
- Define the Job (Jobs API)
1.1. Create a job or use an existing one which isn’t currently being processed with another Batch (sync operation).
2. Define the Batch (Job Batches API)
2.1. Create a new Batch for selected Job (sync operation). It’s possible to create a batch which will authorize a Job after processing it automatically.
3. Upload the original/source content (Job Batches API)
3.1. Upload original files into the Batch (sync operation).
4. Wrap up the Batch (Job Batches API)
4.1. Execute a Batch (sync operation).
When all files are uploaded, the client triggers “execute” for the batch. The batch execution process waits until all files are uploaded and added to the Job. Then, if it was requested, authorizes the Job. The client can check the batch status to see if all files uploaded and attached successfully, view the errors that occurred for each file, and the statistics for added/skipped strings.
As you can see, there are no async calls to the APIs and all the heavy lifting is handled by the service itself.
Conclusion
It’s recommended to use the Job Batches API for third-party content provider integrations as it provides the following benefits, as opposed to the direct approach (File API + Jobs API):
- Requires less time and less steps for the client.
- The Job Batches service is a common entry point for integrations as it handles all the async processing and error handling on its own.
