Home

JSON file encoding

From RFC 4627:

JSON text SHALL be encoded in Unicode. The default encoding is UTF-8.

Since the first two characters of a JSON text will always be ASCII characters, it is possible to determine whether an octet stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking at the pattern of nulls in the first four octets.

 00 00 00 xx  UTF-32BE
 00 xx 00 xx  UTF-16BE
 xx 00 00 00  UTF-32LE
 xx 00 xx 00  UTF-16LE
 xx xx xx xx  UTF-8

---

jsonunicode for Java

JSON documents are generally encoded in UTF-8. For some sets of character data, UTF-16 documents are smaller.

Some JSON APIs support parsing from byte streams. Others only work from character data and rely on the developer to handle character encoding concerns.

jsonunicode is a small Java API for determining the character encoding of an arbitrary JSON document. The JSON specification defines how character encoding is handled.

Situations you may find this API useful are when working with:

• Gson
• JSON.simple
• The Android org.json package

---

Resources

• Home page
  -- javadoc

jsonunicode-1.0.jar is available from Maven central; add this dependency to your pom file:

<dependency>
  <groupId>net.sf.jsonunicode</groupId>
  <artifactId>jsonunicode</artifactId>
  <version>1.0</version>
</dependency>

---

Sample code

{"value":1}

All the sample classes below read a JSON document of the above form, increment the value by one, and write the result.

Gson

Command-line application:

import com.google.gson.Gson;
import net.sf.jsonunicode.JsonIo;
import java.io.*;

/**
 * Gson usage demo (Java 7 syntax)
 */
public class GsonIncrement {
  public static void main(String... args) throws IOException {
    File source = new File(args[0]);
    File destination = new File(args[1]);
    Data data = read(source);
    data.value++;
    write(data, destination);
  }

  public static Data read(File source) throws IOException {
    try (InputStream in = new FileInputStream(source);
         Reader reader = JsonIo.newJsonReader(in)) {
      return new Gson().fromJson(reader, Data.class);
    }
  }

  public static void write(Data data, File destination) throws IOException {
    try (OutputStream out = new FileOutputStream(destination);
         Writer writer = JsonIo.newJsonWriter(out)) {
      new Gson().toJson(data, writer);
    }
  }

  public static class Data {
    public int value;
  }
}

JSON.simple

Command-line application:

import net.sf.jsonunicode.JsonIo;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
import java.io.*;

/**
 * JSON.simple usage demo (Java 7 syntax)
 */
public class SimpleIncrement {
  public static void main(String... args) throws IOException {
    File source = new File(args[0]);
    File destination = new File(args[1]);
    JSONObject data = read(source);
    Number value = (Number) data.get("value");
    data.put("value", value.doubleValue() + 1);
    write(data, destination);
  }

  public static JSONObject read(File source) throws IOException {
    try (InputStream in = new FileInputStream(source);
         Reader reader = JsonIo.newJsonReader(in)) {
      return (JSONObject) new JSONParser().parse(reader);
    } catch (ParseException e) {
      throw new IOException(e);
    }
  }

  public static void write(JSONObject data, File destination) throws IOException {
    try (OutputStream out = new FileOutputStream(destination);
         Writer writer = JsonIo.newJsonWriter(out)) {
      data.writeJSONString(writer);
    }
  }
}

org.json

Load, update + save same file:

import net.sf.jsonunicode.JsonIo;
import org.json.JSONObject;
import org.json.JSONTokener;
import java.io.*;

/**
 * JSON.org API usage demo (Java 6 syntax)
 */
public class JsonDroid {
  public static void increment(File json) throws IOException {
    JSONObject data = read(json);
    Number value = (Number) data.get("value");
    data.put("value", value.doubleValue() + 1);
    write(data, json);
  }

  public static JSONObject read(File source) throws IOException {
    InputStream in = new FileInputStream(source);
    Closeable res = in;
    try {
      Reader reader = JsonIo.newJsonReader(in);
      res = reader;
      return (JSONObject) new JSONTokener(all(reader)).nextValue();
    } finally {
      res.close();
    }
  }

  public static void write(JSONObject data, File destination) throws IOException {
    OutputStream out = new FileOutputStream(destination);
    Closeable res = out;
    try {
      Writer writer = JsonIo.newJsonWriter(out);
      res = writer;
      writer.write(data.toString());
    } finally {
      res.close();
    }
  }

  private static String all(Reader from) throws IOException {
    StringBuilder result = new StringBuilder();
    char[] buffer = new char[1024];
    int len;
    while ((len = from.read(buffer)) != -1) {
      result.append(buffer, 0, len);
    }
    return result.toString();
  }
}

---

Example errors that may indicate this library is required

Trying to decode with the incorrect encoding may result in errors of the following types.

Gson

com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1
  at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176)
  at com.google.gson.Gson.fromJson(Gson.java:803)
  at com.google.gson.Gson.fromJson(Gson.java:741)
  <YOUR CODE>
Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1
  at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:374)
  at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165)
... X more

JSON.simple

Caused by: Unexpected character (?) at position 0.
  at org.json.simple.parser.Yylex.yylex(Unknown Source)
  at org.json.simple.parser.JSONParser.nextToken(Unknown Source)
  at org.json.simple.parser.JSONParser.parse(Unknown Source)
  at org.json.simple.parser.JSONParser.parse(Unknown Source)
  <YOUR CODE>

org.json

org.json.JSONException: Missing value at 0 [character 1 line 1]
  at org.json.JSONTokener.syntaxError(JSONTokener.java:433)
  at org.json.JSONTokener.nextValue(JSONTokener.java:387)
  <YOUR CODE>