.NET SDK

IntoSQL wraps the IntoSQL API for C# and .NET, one of the official SDKs. One call uploads a file, starts the conversion and waits for the result. The package has no dependencies beyond the base library and uses HttpClient, System.Text.Json and System.Security.Cryptography.

Pro: The API is part of IntoSQL Pro. Make an API key in Settings, API keys and keep it out of your source code.

Install#

dotnet add package IntoSQL

The package targets .NET 8 or newer. Every method that calls the API is asynchronous and takes an optional CancellationToken as its last argument.

Create a client#

using IntoSQL;

using var intosql = new IntoSQLClient(Environment.GetEnvironmentVariable("INTOSQL_API_KEY")!);

The constructor throws ArgumentException if the API key is empty. For more options, pass an IntoSQLClientOptions:

using var intosql = new IntoSQLClient(new IntoSQLClientOptions
{
    ApiKey = Environment.GetEnvironmentVariable("INTOSQL_API_KEY")!,
    MaxRetries = 4,
    Timeout = TimeSpan.FromSeconds(30),
});

The client never sends your key to another origin. If a pagination link points somewhere other than BaseUrl's origin, the call throws InvalidOperationException instead.

IntoSQLClient is safe to share between threads; make one and keep it. Disposing it disposes the HttpClient it made, never one you passed. If you pass your own HttpClient, its Timeout also applies to the upload, so set it to Timeout.InfiniteTimeSpan for large files.

Convert a file#

ConvertAsync(path, options) does the whole flow and returns the finished conversion.

using IntoSQL;

using var intosql = new IntoSQLClient(Environment.GetEnvironmentVariable("INTOSQL_API_KEY")!);

var conversion = await intosql.ConvertAsync("ventas.xlsx", new ConvertOptions
{
    Spec = JobSpec.Create(Dialect.PostgreSql, table: "ventas"),
});

Console.WriteLine($"{conversion.Report?.Rows.Converted} {conversion.Output?.Url}");

What happens, in order:

1. Open. POST /conversions with name, the file's size in bytes and spec. The answer has an upload address.
2. Upload. The file is sent with a PUT to that address, with the headers the API gave, as it is. Nothing is read into memory first.
3. Start. POST /conversions/{id}/start.
4. Wait. The conversion is read until its status is done or failed.

The first argument is a path or an Upload. A path is named after the file. For anything else, make an Upload and set Name:

Streams are read but not disposed.

using var http = new HttpClient();
using var response = await http.GetAsync(
    "https://example.com/exports/pedidos.json",
    HttpCompletionOption.ResponseHeadersRead);
await using var stream = await response.Content.ReadAsStreamAsync();

var conversion = await intosql.ConvertAsync(
    Upload.FromStream(stream, response.Content.Headers.ContentLength!.Value),
    new ConvertOptions { Name = "pedidos.json", Spec = JobSpec.Create(Dialect.Sqlite) });

Name helps tell the file's format, and the downloaded result is named after it: ventas.xlsx downloads as ventas.sql. Set table in the spec to choose the table's name.

Database dumps#

A MySQL or MariaDB dump (.sql, .sql.gz) converts to PostgreSQL like any file. Its report lists each table in Tables, where the dump came from in Source, and the views, triggers and routines it did not carry in NotConverted:

foreach (var table in conversion.Report!.Tables)
{
    Console.WriteLine($"{table.Name}: {table.Rows} rows");
}

How it waits#

ConvertAsync, ConvertUrlAsync and Conversions.WaitAsync read the conversion again and again with growing pauses: half a second, then 1, 2 and 4 seconds, then every 5 seconds. They return when the status is done, throw ConversionFailedException when it is failed, and ConversionCanceledException when it was canceled.

Giving up only stops the waiting. The conversion keeps running on IntoSQL's servers, and you can read it later by its Id.

using var stop = new CancellationTokenSource(TimeSpan.FromMinutes(15));

var conversion = await intosql.ConvertAsync(
    "clientes_2024.xlsx",
    new ConvertOptions
    {
        Spec = JobSpec.Create(Dialect.MySql),
        Timeout = TimeSpan.FromMinutes(10),
    },
    stop.Token);

Convert a file by URL#

ConvertUrlAsync(url, options) asks IntoSQL to download the file from a public HTTPS address, then waits the same way. There is no upload step.

var conversion = await intosql.ConvertUrlAsync(
    "https://example.com/exports/ventas.csv",
    new ConvertOptions { Spec = JobSpec.Create(Dialect.MySql) });

ConvertOptions takes Spec, an optional Name (otherwise the name comes from the server's Content-Disposition header or the address), and Timeout. The address must be HTTPS and reachable from the internet; anything else is refused with invalid_url. If the download fails, the conversion ends as failed.

Download the result#

When Status is done, Output.Url downloads the result. The address works for ten minutes. After that, read the conversion again to get a fresh one.

var fresh = await intosql.Conversions.RetrieveAsync(conversion.Id);

using var http = new HttpClient();
await using var download = await http.GetStreamAsync(fresh.Output!.Url);
await using var file = File.Create("ventas.sql");
await download.CopyToAsync(file);

Send no Authorization header to Output.Url; the address carries its own signature. Files are deleted 24 hours after the conversion finishes, or right away with Conversions.DeleteAsync.

Build a spec#

JobSpec.Create(dialect, ...) returns a full JobSpec with IntoSQL's defaults. Only dialect is required; the rest are named arguments.

var request = JobSpec.Create(
    Dialect.SqlServer,
    table: "clientes",
    schema: "ventas",
    sheet: "Clientes 2024",
    batchRows: 1000);

Dialect has PostgreSql, MySql, MariaDb, Sqlite, SqlServer and Oracle; OutputFormat has Sql, Csv and Json.

The rest is fixed by the helper: format, encoding, delimiter and header row detected from the file; column names in snake_case; INSERT statements (Style = "inserts"); types inferred from the first 10,000 rows; date order detected; leading zeros kept; no per-column overrides. For anything else (bulk output, a date order, a locale, columns renamed, kept as text or left out), change the object it returns. Every field is in Conversion spec.

var request = JobSpec.Create(Dialect.PostgreSql, table: "pedidos");

request.Output.Style = "bulk";
request.Inference.DateOrder = "day_first";
request.Columns.Add(new ColumnSpec { Index = 0, AsText = true });

Conversions#

intosql.Conversions exposes each step on its own, for when you want to upload from somewhere else, start later, or not wait.

Step by step#

var file = Upload.FromFile("ventas.xlsx");

var opened = await intosql.Conversions.CreateAsync("ventas.xlsx", file.Bytes, JobSpec.Create(Dialect.MariaDb));

await intosql.Conversions.UploadAsync(opened, file);
await intosql.Conversions.StartAsync(opened.Id);

// Later, or from a webhook handler:
var conversion = await intosql.Conversions.RetrieveAsync(opened.Id);

The upload address from CreateAsync works for an hour. UploadAsync throws InvalidOperationException if the conversion has no upload address (one read with RetrieveAsync never does), and throws an IntoSQLException with code upload_failed if storage doesn't answer with a 2xx.

List#

ListAsync() yields every conversion on the account, newest first. It fetches a page at a time as you enumerate (50, or limit from 1 to 100) and follows links.next until there are no more pages. status keeps only conversions in that status.

await foreach (var conversion in intosql.Conversions.ListAsync(status: ConversionStatus.Failed))
{
    Console.WriteLine($"{conversion.Id} {conversion.Error?.Code} {conversion.Name}");
}

Stop early with break; no further pages are fetched.

Cancel#

CancelAsync(id) stops a conversion that hasn't ended and deletes its files. It stays listed as canceled and doesn't count toward the month. Canceling it again answers the same; canceling one that already ended throws IntoSQLException with code already_finished.

await intosql.Conversions.CancelAsync(conversion.Id);

Send us a conversion#

When a file didn't convert well, or at all, CreateCaseAsync sends the conversion to IntoSQL so support for it gets added. Set Consent = true only when the person agreed to send it. Share = "file" sends the file as well as the report; a failed conversion keeps its upload until it expires, so the file is still there to send.

await intosql.Conversions.CreateCaseAsync(conversion.Id, new CreateCaseOptions
{
    Consent = true,
    Share = "file",
    Note = "Exported from our ERP; the dates come out wrong.",
});

CreateCaseOptions also takes Email, where to write back about it. Cases are private, kept 90 days, and then deleted. The privacy policy says how they are handled.

Delete#

DeleteAsync(id) removes the conversion's input and output files now instead of after 24 hours. Reading it afterwards answers not_found.

await intosql.Conversions.DeleteAsync(conversion.Id);

Usage#

GetUsageAsync() returns the bytes the plan includes this month, used and left.

var usage = await intosql.GetUsageAsync();

Console.WriteLine($"{usage.Used} of {usage.Limit} bytes used since {usage.PeriodStart:d}; {usage.Left} left.");

Failed conversions don't count toward Used.

Verify webhooks#

With a webhook address saved in Settings, API keys, IntoSQL sends a signed POST when a conversion is done, failed or canceled. Webhooks.Parse checks the IntoSQL-Signature header against the raw body and returns the typed event; it throws WebhookSignatureException when the signature doesn't match.

using IntoSQL;

app.MapPost("/webhooks/intosql", async (HttpRequest request) =>
{
    using var reader = new StreamReader(request.Body);
    var body = await reader.ReadToEndAsync();
    WebhookEvent webhookEvent;

    try
    {
        webhookEvent = Webhooks.Parse(
            body,
            request.Headers["IntoSQL-Signature"].ToString(),
            Environment.GetEnvironmentVariable("INTOSQL_WEBHOOK_SECRET")!);
    }
    catch (WebhookSignatureException)
    {
        return Results.BadRequest();
    }

    if (webhookEvent.Type == WebhookEvent.ConversionDone)
    {
        // webhookEvent.Conversion is the conversion; webhookEvent.Id is the same on every retry.
    }

    return Results.NoContent();
});

Webhooks.Verify(body, header, secret) does the check alone and returns true or false. Both refuse a call when the header is malformed, the signature doesn't match, or the timestamp is more than 300 seconds from now. The named arguments tolerance (a TimeSpan) change that window and now (a DateTimeOffset) fixes the current time in tests. Pass the body exactly as received, before any JSON parsing. The payload and retries are in Webhooks.

Errors#

The SDK throws these exceptions.

• IntoSQLException: the API answered with an error. It has Status (the HTTP status), Code (stable between versions), Message (for people), Fields (per field messages when the code is invalid_request, otherwise empty) and RequestId, to quote when you write to us. If the answer has no error body, Code is http_error.
• ConversionFailedException: the conversion ended as failed. It has Id, Code (stable: ours, such as timed_out, or the engine's, such as encrypted_file), Problems (what the engine found) and the explanation as its Message.
• ConversionCanceledException: the conversion was canceled while it was awaited. It has Id.
• WebhookSignatureException: Webhooks.Parse got a call that didn't check out.

A connection that keeps failing after the retries throws the HttpRequestException from HttpClient, and an API that keeps not answering within Timeout throws a TimeoutException. Waiting past the wait Timeout throws a TimeoutException too. Canceling the CancellationToken throws an OperationCanceledException.

try
{
    await intosql.ConvertAsync("ventas.xlsx", new ConvertOptions { Spec = JobSpec.Create(Dialect.Oracle) });
}
catch (ConversionFailedException error)
{
    Console.Error.WriteLine($"Conversion {error.Id} failed ({error.Code}): {error.Message}");
}
catch (IntoSQLException error) when (error.Code == "over_quota")
{
    Console.Error.WriteLine("This month's conversions are used up.");
}
catch (IntoSQLException error)
{
    Console.Error.WriteLine($"{error.Status} {error.Code} {error.Message} {error.RequestId}");
}

Every code, its status and how to fix it is in Errors.

Retries and idempotency#

The client retries a request when the connection fails or the API doesn't answer within Timeout, when the API answers rate_limited, too_many_active or request_in_progress, on any 5xx, and on 408 and 425 from the network in between, up to MaxRetries times (2 by default). It waits for Retry-After when the answer has it, and otherwise half a second, then 1 second, doubling up to 8 seconds.

Every POST gets an Idempotency-Key header with a random UUID, and its retries reuse the same key. If a first attempt reached the API but its answer was lost, the retry gets that first answer back instead of opening a second conversion.

Not retried: the file upload to storage (a failed PUT throws upload_failed), already_started (the conversion is on its way; read it instead), and any other 4xx such as invalid_spec or over_quota.

Every request also sends User-Agent: intosql-dotnet/<version>; the version is IntoSQLClient.Version.

Types#

The package has these types alongside the client and the exceptions, all in the IntoSQL namespace:

Properties are PascalCase in C# and snake_case in the JSON: CreatedAt is created_at. Dialect, OutputFormat and ConversionStatus compare by their API name and convert from a string, so conversion.Status == "done" works. Codes and other values the engine may add to (Problem.Code, Fix.Code, ColumnReport.Kind, NotConverted.Kind, OutputSpec.Style) are plain strings. Report is described in Conversions, and fixes in Fixes.