Go SDK

intosql wraps the IntoSQL API for Go, one of the official SDKs. One call uploads a file, starts the conversion and waits for the result. The package uses only the standard library: net/http, encoding/json and crypto/hmac.

Pro: The API is part of IntoSQL Pro. Make an API key in Settings, API keys and keep it out of your source code.

Install#

go get github.com/alfonsobries/intosql-go

The module needs Go 1.22 or newer. Import it as github.com/alfonsobries/intosql-go; the package is named intosql.

Create a client#

import "github.com/alfonsobries/intosql-go"

client, err := intosql.NewClient(os.Getenv("INTOSQL_API_KEY"))
if err != nil {
	log.Fatal(err)
}

NewClient returns an error if the API key is an empty string or the base URL isn't an absolute address. Options go after the key:

client, err := intosql.NewClient(
	os.Getenv("INTOSQL_API_KEY"),
	intosql.WithMaxRetries(5),
	intosql.WithTimeout(30*time.Second),
)

Every method takes a context.Context first. Canceling it stops the request, the retries and the waiting. The client is safe for concurrent use.

The client never sends your key to another origin. If a pagination link points somewhere other than the base URL's origin, the call returns an error instead.

Convert a file#

ConvertFile(ctx, path, options) does the whole flow and returns the finished conversion.

conversion, err := client.ConvertFile(ctx, "ventas.xlsx", intosql.ConvertOptions{
	Spec: intosql.NewSpec(intosql.SpecOptions{Dialect: intosql.DialectPostgreSQL, Table: "ventas"}),
})
if err != nil {
	log.Fatal(err)
}

fmt.Println(conversion.Report.Rows.Converted, conversion.Output.URL)

What happens, in order:

1. Open. POST /conversions with name, the file's size in bytes and spec. The answer has an upload address.
2. Upload. The file is sent with a PUT to that address, with the headers the API gave, as it is. Nothing is read into memory first.
3. Start. POST /conversions/{id}/start.
4. Wait. The conversion is read until its status is done or failed.

ConvertFile names the conversion after the file (ventas.xlsx) unless Name is set. Convert(ctx, reader, size, options) takes any io.Reader with its size in bytes instead. The size must be exact, since the conversion only starts once the whole file has arrived.

response, err := http.Get("https://example.com/exports/pedidos.json")
if err != nil {
	log.Fatal(err)
}
defer response.Body.Close()

conversion, err := client.Convert(ctx, response.Body, response.ContentLength, intosql.ConvertOptions{
	Name: "pedidos.json",
	Spec: intosql.NewSpec(intosql.SpecOptions{Dialect: intosql.DialectSQLite}),
})

Name helps tell the file's format, and the downloaded result is named after it: ventas.xlsx downloads as ventas.sql. Set Table in the spec to choose the table's name.

A MySQL or MariaDB dump (.sql, .sql.gz) converts to PostgreSQL like any file. The report then lists each table in Tables, with its own columns, fixes and problems, and the views, triggers and routines it did not carry in NotConverted.

for _, table := range conversion.Report.Tables {
	fmt.Println(table.Name, table.Rows)
}

How it waits#

Convert, ConvertFile, ConvertURL and Conversions.Wait read the conversion again and again with growing pauses: half a second, then 1, 2 and 4 seconds, then every 5 seconds. They return the conversion when the status is done, *ConversionFailedError when it is failed, and *ConversionCanceledError when it was canceled.

To stop waiting early, cancel the context; the context's error is returned. Giving up only stops the waiting. The conversion keeps running on IntoSQL's servers, and you can read it later by its ID.

ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
defer cancel()

conversion, err := client.ConvertFile(ctx, "clientes_2024.xlsx", intosql.ConvertOptions{
	Spec: intosql.NewSpec(intosql.SpecOptions{Dialect: intosql.DialectMySQL}),
	Wait: intosql.WaitOptions{Timeout: 10 * time.Minute},
})

Convert a file by URL#

ConvertURL(ctx, url, options) asks IntoSQL to download the file from a public HTTPS address, then waits the same way. There is no upload step.

conversion, err := client.ConvertURL(ctx, "https://example.com/exports/ventas.csv", intosql.ConvertURLOptions{
	Spec: intosql.NewSpec(intosql.SpecOptions{Dialect: intosql.DialectMySQL}),
})

ConvertURLOptions takes Spec, an optional Name (otherwise the name comes from the server's Content-Disposition header or the address), and Wait. The address must be HTTPS and reachable from the internet; anything else is refused with invalid_url. If the download fails, the conversion ends as failed.

Download the result#

When Status is done, Output.URL downloads the result. The address works for ten minutes. After that, read the conversion again to get a fresh one.

fresh, err := client.Conversions.Retrieve(ctx, conversion.ID)
if err != nil {
	log.Fatal(err)
}

download, err := http.Get(fresh.Output.URL)
if err != nil {
	log.Fatal(err)
}
defer download.Body.Close()

file, err := os.Create("ventas.sql")
if err != nil {
	log.Fatal(err)
}
defer file.Close()

if _, err := io.Copy(file, download.Body); err != nil {
	log.Fatal(err)
}

Send no Authorization header to Output.URL; the address carries its own signature. Files are deleted 24 hours after the conversion finishes, or right away with Conversions.Delete.

Build a spec#

NewSpec(options) returns a full JobSpec with IntoSQL's defaults. Only Dialect is required; a zero value takes the default.

request := intosql.NewSpec(intosql.SpecOptions{
	Dialect:   intosql.DialectSQLServer,
	Table:     "clientes",
	Schema:    "ventas",
	Sheet:     "Clientes 2024",
	BatchRows: 1000,
})

The dialects are DialectPostgreSQL, DialectMySQL, DialectMariaDB, DialectSQLite, DialectSQLServer and DialectOracle; the formats FormatSQL, FormatCSV and FormatJSON.

The rest is fixed by the helper: format, encoding, delimiter and header row detected from the file; column names in snake_case; INSERT statements (StyleInserts); types inferred from the first 10,000 rows; date order detected; leading zeros kept; no per-column overrides. For anything else (bulk output, a date order, a locale, columns renamed, kept as text or left out), change the value it returns or build a JobSpec yourself. Every field is in Conversion spec.

request := intosql.NewSpec(intosql.SpecOptions{Dialect: intosql.DialectPostgreSQL, Table: "pedidos"})

request.Output.Style = intosql.StyleBulk
request.Inference.DateOrder = intosql.DateOrderDayFirst
request.Columns = []intosql.ColumnSpec{{Index: 0, AsText: true}}

Conversions#

client.Conversions exposes each step on its own, for when you want to upload from somewhere else, start later, or not wait.

Each method also returns an error.

Step by step#

file, err := os.Open("ventas.xlsx")
if err != nil {
	log.Fatal(err)
}
defer file.Close()

info, _ := file.Stat()

opened, err := client.Conversions.Create(ctx, intosql.CreateParams{
	Name:  "ventas.xlsx",
	Bytes: info.Size(),
	Spec:  intosql.NewSpec(intosql.SpecOptions{Dialect: intosql.DialectMariaDB}),
})
if err != nil {
	log.Fatal(err)
}

if err := client.Conversions.Upload(ctx, opened, file, info.Size()); err != nil {
	log.Fatal(err)
}

if _, err := client.Conversions.Start(ctx, opened.ID); err != nil {
	log.Fatal(err)
}

// Later, or from a webhook handler:
conversion, err := client.Conversions.Retrieve(ctx, opened.ID)

The upload address from Create works for an hour. Upload returns an error if the conversion has no upload address (one read with Retrieve never does), and an *Error with code upload_failed if storage doesn't answer with a 2xx. It sends the size as Content-Length, so storage gets one plain PUT.

List#

List walks every conversion on the account, newest first. It fetches a page at a time (50, or Limit from 1 to 100) and follows links.next until there are no more pages. Status keeps only conversions in that status.

conversions := client.Conversions.List(ctx, intosql.ListParams{Status: intosql.StatusFailed})
for conversions.Next() {
	conversion := conversions.Current()
	fmt.Println(conversion.ID, conversion.Error.Code, conversion.Name)
}

if err := conversions.Err(); err != nil {
	log.Fatal(err)
}

Nothing is fetched until the first Next. Stop early with break; no further pages are fetched. Next returns false at the end of the list and on an error, so check Err after the loop.

Cancel#

Cancel(ctx, id) stops a conversion that hasn't ended and deletes its files. It stays listed as canceled and doesn't count toward the month. Canceling it again answers the same; canceling one that already ended returns *Error with code already_finished.

_, err := client.Conversions.Cancel(ctx, conversion.ID)

Send us a conversion#

When a file didn't convert well, or at all, CreateCase sends the conversion to IntoSQL so support for it gets added. Set Consent: true only when the person agreed to send it. Share: intosql.ShareFile sends the file as well as the report; a failed conversion keeps its upload until it expires, so the file is still there to send.

_, err := client.Conversions.CreateCase(ctx, conversion.ID, intosql.CaseParams{
	Consent: true,
	Share:   intosql.ShareFile,
	Note:    "Exported from our ERP; the dates come out wrong.",
})

Cases are private, kept 90 days, and then deleted. The privacy policy says how they are handled.

Delete#

Delete(ctx, id) removes the conversion's input and output files now instead of after 24 hours. Reading it afterwards answers not_found.

err := client.Conversions.Delete(ctx, conversion.ID)

Usage#

Usage(ctx) returns the bytes the plan includes this month, used and left.

usage, err := client.Usage(ctx)
if err != nil {
	log.Fatal(err)
}

fmt.Println(usage.Limit, usage.Used, usage.Left, usage.PeriodStart)

Failed conversions don't count toward Used.

Verify webhooks#

With a webhook address saved in Settings, API keys, IntoSQL sends a signed POST when a conversion is done, failed or canceled. ParseWebhook checks the IntoSQL-Signature header against the raw body and returns the typed event; it returns ErrInvalidSignature when the signature doesn't match.

http.HandleFunc("/webhooks/intosql", func(writer http.ResponseWriter, request *http.Request) {
	body, err := io.ReadAll(request.Body)
	if err != nil {
		writer.WriteHeader(http.StatusBadRequest)
		return
	}

	event, err := intosql.ParseWebhook(body, request.Header.Get("IntoSQL-Signature"), os.Getenv("INTOSQL_WEBHOOK_SECRET"))
	if err != nil {
		writer.WriteHeader(http.StatusBadRequest)
		return
	}

	if event.Type == intosql.EventConversionDone {
		// event.Conversion is the conversion; event.ID is the same on every retry.
	}

	writer.WriteHeader(http.StatusNoContent)
})

VerifyWebhook(body, header, secret) does the check alone and returns true or false. Both refuse a call when the header is malformed, the signature doesn't match, or the timestamp is more than 300 seconds from now. intosql.WebhookTolerance(duration) changes that window and intosql.WebhookNow(time) fixes the current time in tests; pass them after the secret. Pass the body exactly as received, before any JSON parsing. The payload and retries are in Webhooks.

WebhookEvent has ID, Type, CreatedAt and Data, the raw JSON. For conversion.done, conversion.failed and conversion.canceled, Conversion holds the decoded conversion; for ping, Ping holds SentAt. An event type added later comes with Data alone.

Errors#

The SDK returns typed errors; tell them apart with errors.As.

• *intosql.Error: the API answered with an error. It has Status (the HTTP status), Code (stable between versions), Message (for people), Fields (per field messages when the code is invalid_request, otherwise empty) and RequestID, to quote when you write to us. If the answer has no error body, Code is http_error.
• *intosql.ConversionFailedError: the conversion ended as failed. It has ID, Code (stable: ours, such as timed_out, or the engine's, such as encrypted_file), Problems (what the engine found) and the explanation as its Message.
• *intosql.ConversionCanceledError: the conversion was canceled while it was awaited. It has ID.
• *intosql.WaitTimeoutError: waiting went past Timeout. It has ID and the last Status.

A connection that keeps failing after the retries returns the error from net/http. A canceled context returns the context's error.

_, err := client.ConvertFile(ctx, "ventas.xlsx", intosql.ConvertOptions{
	Spec: intosql.NewSpec(intosql.SpecOptions{Dialect: intosql.DialectOracle}),
})

var failed *intosql.ConversionFailedError
var apiError *intosql.Error

switch {
case errors.As(err, &failed):
	log.Printf("Conversion %s failed (%s): %s", failed.ID, failed.Code, failed.Message)
case errors.As(err, &apiError) && apiError.Code == "over_quota":
	log.Print("This month's conversions are used up.")
case errors.As(err, &apiError):
	log.Print(apiError.Status, apiError.Code, apiError.Message, apiError.Fields)
case err != nil:
	log.Fatal(err)
}

Every code, its status and how to fix it is in Errors.

Retries and idempotency#

The client retries a request when the connection fails, when the API answers rate_limited, too_many_active or request_in_progress, on any 5xx, and on 408 and 425 from the network in between, up to WithMaxRetries times (2 by default). It waits for Retry-After when the answer has it, and otherwise half a second, then 1 second, doubling up to 8 seconds. It stops at once when the context ends.

Every POST gets an Idempotency-Key header with a random UUID, and its retries reuse the same key. If a first attempt reached the API but its answer was lost, the retry gets that first answer back instead of opening a second conversion.

Not retried: the file upload to storage (a failed PUT returns upload_failed), already_started (the conversion is on its way; read it instead), and any other 4xx such as invalid_spec or over_quota.

Every request sends User-Agent: intosql-go/<version>.

Types#

The package exports these types alongside the client and functions:

Nullable fields are pointers, and enum types are strings, so values added to the API later still decode. Report is described in Conversions, and fixes in Fixes.