Java SDK

com.intosql:intosql wraps the IntoSQL API for Java and other JVM languages, one of the official SDKs. One call uploads a file, starts the conversion and waits for the result. The package uses the JDK's java.net.http.HttpClient and depends only on Jackson.

Pro: The API is part of IntoSQL Pro. Make an API key in Settings, API keys and keep it out of your source code.

Install#

With Maven:

<dependency>
    <groupId>com.intosql</groupId>
    <artifactId>intosql</artifactId>
    <version>0.2.0</version>
</dependency>

With Gradle:

implementation("com.intosql:intosql:0.2.0")

The package runs on Java 11 or newer. Its module name is com.intosql.

Create a client#

import com.intosql.IntoSQL;

IntoSQL intosql = new IntoSQL(System.getenv("INTOSQL_API_KEY"));

The constructor throws IllegalArgumentException if the API key is null or empty. For other options, use the builder:

import java.time.Duration;

IntoSQL intosql = IntoSQL.builder()
        .apiKey(System.getenv("INTOSQL_API_KEY"))
        .maxRetries(4)
        .timeout(Duration.ofSeconds(30))
        .build();

The client is thread safe; make one and share it. It never sends your key to another origin. If a pagination link points somewhere other than baseUrl's origin, the call throws IllegalStateException instead.

To go through a proxy, pass your own HttpClient:

import com.intosql.JavaHttpTransport;
import java.net.InetSocketAddress;
import java.net.ProxySelector;
import java.net.http.HttpClient;

IntoSQL intosql = IntoSQL.builder()
        .apiKey(System.getenv("INTOSQL_API_KEY"))
        .transport(new JavaHttpTransport(HttpClient.newBuilder()
                .proxy(ProxySelector.of(new InetSocketAddress("proxy.internal", 3128)))
                .build()))
        .build();

Convert a file#

convert(path, spec) does the whole flow and returns the finished conversion.

import com.intosql.IntoSQL;
import com.intosql.Spec;
import com.intosql.model.Conversion;
import com.intosql.model.Dialect;
import java.nio.file.Path;

IntoSQL intosql = new IntoSQL(System.getenv("INTOSQL_API_KEY"));

Conversion conversion = intosql.convert(
        Path.of("ventas.xlsx"),
        Spec.builder(Dialect.POSTGRESQL).table("ventas").build());

System.out.println(conversion.getReport().getRows().getConverted());
System.out.println(conversion.getOutput().getUrl());

What happens, in order:

1. Open. POST /conversions with name, the file's size in bytes and spec. The answer has an upload address.
2. Upload. The file is sent with a PUT to that address, with the headers the API gave, as it is. Nothing is read into memory first.
3. Start. POST /conversions/{id}/start.
4. Wait. The conversion is read until its status is done or failed.

With a Path, the file's name is sent as name. For anything else, pass an Upload and the name: Upload.of(path), Upload.of(bytes) for a byte[], or Upload.of(stream, bytes) for an InputStream with its size in bytes. The size must be exact, since the conversion only starts once the whole file has arrived. A stream is read once.

import com.intosql.Upload;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.io.InputStream;

HttpResponse<InputStream> response = HttpClient.newHttpClient().send(
        HttpRequest.newBuilder(URI.create("https://example.com/exports/pedidos.json")).build(),
        HttpResponse.BodyHandlers.ofInputStream());

Conversion conversion = intosql.convert(
        Upload.of(response.body(), response.headers().firstValueAsLong("Content-Length").orElseThrow()),
        "pedidos.json",
        Spec.of(Dialect.SQLITE));

name helps tell the file's format, and the downloaded result is named after it: ventas.xlsx downloads as ventas.sql. Set table in the spec to choose the table's name.

A MySQL or MariaDB dump (.sql, .sql.gz) converts to PostgreSQL like any file. Its report lists each table with getTables(), and what it did not carry with getNotConverted():

for (TableReport table : conversion.getReport().getTables()) {
    System.out.println(table.getName() + ": " + table.getRows() + " rows");
}

How it waits#

convert, convertUrl and conversions().waitFor read the conversion again and again with growing pauses: half a second, then 1, 2 and 4 seconds, then every 5 seconds. They return when the status is done, throw ConversionFailedException when it is failed, and ConversionCanceledException when it was canceled.

Interrupting the thread stops the wait: it throws java.util.concurrent.CancellationException and keeps the thread's interrupt flag set. Giving up only stops the waiting. The conversion keeps running on IntoSQL's servers, and you can read it later by its id.

import java.time.Duration;

Conversion conversion = intosql.convert(
        Path.of("clientes_2024.xlsx"),
        Spec.of(Dialect.MYSQL),
        Duration.ofMinutes(10));

Convert a file by URL#

convertUrl(url, spec) asks IntoSQL to download the file from a public HTTPS address, then waits the same way. There is no upload step.

Conversion conversion = intosql.convertUrl(
        "https://example.com/exports/ventas.csv",
        Spec.of(Dialect.MYSQL));

convertUrl(url, spec, name, timeout) also takes a name (null takes it from the server's Content-Disposition header or the address) and the wait's timeout. The address must be HTTPS and reachable from the internet; anything else is refused with invalid_url. If the download fails, the conversion ends as failed.

Download the result#

When the status is done, getOutput().getUrl() downloads the result. The address works for ten minutes. After that, read the conversion again to get a fresh one.

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

Conversion fresh = intosql.conversions().retrieve(conversion.getId());

HttpClient.newHttpClient().send(
        HttpRequest.newBuilder(URI.create(fresh.getOutput().getUrl())).build(),
        HttpResponse.BodyHandlers.ofFile(Path.of("ventas.sql")));

Send no Authorization header to the output address; it carries its own signature. Files are deleted 24 hours after the conversion finishes, or right away with conversions().delete.

Build a spec#

Spec.of(dialect) returns a full JobSpec with IntoSQL's defaults. Spec.builder(dialect) sets the options below first. Only the dialect is required.

import com.intosql.Spec;
import com.intosql.model.Dialect;
import com.intosql.model.JobSpec;

JobSpec request = Spec.builder(Dialect.SQL_SERVER)
        .table("clientes")
        .schema("ventas")
        .sheet("Clientes 2024")
        .batchRows(1000)
        .build();

Dialect is POSTGRESQL, MYSQL, MARIADB, SQLITE, SQL_SERVER or ORACLE. OutputFormat is SQL, CSV or JSON.

The rest is fixed by the helper: format, encoding, delimiter and header row detected from the file; column names in snake_case; INSERT statements (OutputStyle.INSERTS); types inferred from the first 10,000 rows; date order detected; leading zeros kept; no per-column overrides. For anything else (bulk output, a date order, a locale, columns renamed, kept as text or left out), change the object it returns with its setters, or build a JobSpec yourself. Every field is in Conversion spec.

import com.intosql.model.ColumnSpec;
import com.intosql.model.DateOrder;
import com.intosql.model.OutputStyle;
import java.util.List;

JobSpec request = Spec.builder(Dialect.POSTGRESQL).table("pedidos").build();

request.getOutput().setStyle(OutputStyle.BULK);
request.getInference().setDateOrder(DateOrder.DAY_FIRST);

ColumnSpec code = new ColumnSpec();
code.setIndex(0);
code.setAsText(true);
request.setColumns(List.of(code));

Conversions#

intosql.conversions() exposes each step on its own, for when you want to upload from somewhere else, start later, or not wait.

Step by step#

import com.intosql.Upload;
import java.nio.file.Files;

Path file = Path.of("ventas.xlsx");

Conversion opened = intosql.conversions().create(
        "ventas.xlsx", Files.size(file), Spec.of(Dialect.MARIADB));

intosql.conversions().upload(opened, Upload.of(file));
intosql.conversions().start(opened.getId());

// Later, or from a webhook handler:
Conversion conversion = intosql.conversions().retrieve(opened.getId());

The upload address from create works for an hour. upload throws IllegalStateException if the conversion has no upload address (one read with retrieve never does), and throws an IntoSQLException with code upload_failed if storage doesn't answer with a 2xx.

List#

list() returns every conversion on the account, newest first. It fetches a page at a time as you iterate (50, or limit from 1 to 100) and follows links.next until there are no more pages. status keeps only conversions in that status.

import com.intosql.ListParams;
import com.intosql.model.ConversionStatus;

for (Conversion conversion : intosql.conversions().list(new ListParams().status(ConversionStatus.FAILED))) {
    System.out.println(conversion.getId() + " " + conversion.getError().getCode() + " " + conversion.getName());
}

Stop early with break; no further pages are fetched.

Cancel#

cancel(id) stops a conversion that hasn't ended and deletes its files. It stays listed as canceled and doesn't count toward the month. Canceling it again answers the same; canceling one that already ended throws IntoSQLException with code already_finished.

intosql.conversions().cancel(conversion.getId());

Send us a conversion#

When a file didn't convert well, or at all, createCase sends the conversion to IntoSQL so support for it gets added. Pass consent(true) only when the person agreed to send it. share(CaseShare.FILE) sends the file as well as the report; a failed conversion keeps its upload until it expires, so the file is still there to send.

import com.intosql.CaseParams;
import com.intosql.model.CaseShare;

intosql.conversions().createCase(conversion.getId(), new CaseParams()
        .consent(true)
        .share(CaseShare.FILE)
        .note("Exported from our ERP; the dates come out wrong."));

CaseParams also takes email(...), an address to write back to. Cases are private, kept 90 days, and then deleted. The privacy policy says how they are handled.

Delete#

delete(id) removes the conversion's input and output files now instead of after 24 hours. Reading it afterwards answers not_found.

intosql.conversions().delete(conversion.getId());

Usage#

usage() returns the bytes the plan includes this month, used and left.

import com.intosql.model.Usage;

Usage usage = intosql.usage();
System.out.println(usage.getUsed() + " of " + usage.getLimit() + " since " + usage.getPeriodStart());

Failed conversions don't count toward used.

Verify webhooks#

With a webhook address saved in Settings, API keys, IntoSQL sends a signed POST when a conversion is done, failed or canceled. Webhooks.parse checks the IntoSQL-Signature header against the raw body and returns the event; it throws WebhookSignatureException when the signature doesn't match. With the JDK's own server:

import com.intosql.WebhookEvent;
import com.intosql.WebhookSignatureException;
import com.intosql.Webhooks;
import com.sun.net.httpserver.HttpExchange;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

void handle(HttpExchange exchange) throws IOException {
    String body = new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
    String header = exchange.getRequestHeaders().getFirst("IntoSQL-Signature");
    WebhookEvent event;

    try {
        event = Webhooks.parse(body, header == null ? "" : header, System.getenv("INTOSQL_WEBHOOK_SECRET"));
    } catch (WebhookSignatureException exception) {
        exchange.sendResponseHeaders(400, -1);
        exchange.close();
        return;
    }

    if (WebhookEvent.CONVERSION_DONE.equals(event.getType())) {
        // event.getConversion() is the conversion; event.getId() is the same on every retry.
    }

    exchange.sendResponseHeaders(204, -1);
    exchange.close();
}

The event has getId(), getType() (WebhookEvent.CONVERSION_DONE, CONVERSION_FAILED, CONVERSION_CANCELED or PING), getCreatedAt(), getConversion() for the conversion types, getSentAt() for a ping, and getData() with the raw data as a Jackson JsonNode.

Webhooks.verify(body, header, secret) does the check alone and returns true or false. Both refuse a call when the header is malformed, the signature doesn't match, or the timestamp is more than 300 seconds from now. The overloads with two more arguments take the tolerance as a Duration and the current time as an Instant, to change that window or fix the time in tests. Pass the body exactly as received, before any JSON parsing. The payload and retries are in Webhooks.

Errors#

Every exception the SDK throws is unchecked.

• IntoSQLException: the API answered with an error. It has getStatus() (the HTTP status), getCode() (stable between versions), getMessage() (for people), getFields() (per field messages when the code is invalid_request, otherwise empty) and getRequestId(), to quote when you write to us. If the answer has no error body, the code is http_error.
• ConversionFailedException: the conversion ended as failed. It has getId(), getCode() (stable: ours, such as timed_out, or the engine's, such as encrypted_file), getProblems() (what the engine found) and the explanation as its message.
• ConversionCanceledException: the conversion was canceled while it was awaited. It has getId().
• ConversionTimeoutException: the wait's timeout ran out. It has getId() and getStatus().

A connection that keeps failing after the retries throws java.io.UncheckedIOException, with the transport's IOException as its cause.

import com.intosql.ConversionFailedException;
import com.intosql.IntoSQLException;

try {
    intosql.convert(Path.of("ventas.xlsx"), Spec.of(Dialect.ORACLE));
} catch (ConversionFailedException error) {
    System.err.println("Conversion " + error.getId() + " failed (" + error.getCode() + "): " + error.getMessage());
} catch (IntoSQLException error) {
    if (error.getCode().equals("over_quota")) {
        System.err.println("This month's conversions are used up.");
    } else {
        System.err.println(error.getStatus() + " " + error.getCode() + " " + error.getMessage() + " " + error.getFields());
    }
}

Every code, its status and how to fix it is in Errors.

Retries and idempotency#

The client retries a request when the connection fails, when the API answers rate_limited, too_many_active or request_in_progress, on any 5xx, and on 408 and 425 from the network in between, up to maxRetries times (2 by default). It waits for Retry-After when the answer has it, and otherwise half a second, then 1 second, doubling up to 8 seconds.

Every POST gets an Idempotency-Key header with a random UUID, and its retries reuse the same key. If a first attempt reached the API but its answer was lost, the retry gets that first answer back instead of opening a second conversion.

Not retried: the file upload to storage (a failed PUT throws upload_failed), already_started (the conversion is on its way; read it instead), and any other 4xx such as invalid_spec or over_quota.

Types#

Requests and answers are classes in com.intosql.model, read with getters. Enum values the API adds after this version of the SDK read as UNKNOWN instead of failing.

JobSpec and JobReport follow the engine's contract, so they match what the API accepts and returns. Fix and problem codes are strings, since new ones may be added within v1. report is described in Conversions, and fixes in Fixes.