Monetisation of Susi using the Amazon Product API (Part 2)

So in my previous blog post, I covered about the semantics of the Amazon Product Advertising API, and how does the monetisation work. Today, let’s jump into the code and the relevance with Susi.

We have seen that the Amazon Product Advertising API is a SOAP API. The query of the SOAP API access goes like this:

http://webservices.amazon.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[AWS Access Key ID]&AssociateTag=[Associate ID]&Operation=ItemSearch&Keywords=the%20hunger%20games&SearchIndex=Books&Timestamp=[YYYY-MM-DDThh:mm:ssZ]&Signature=[Request Signature]

We supply to it the Operation (ItemSearch, ItemLookup etc, you can have the full list here), the Keywords to look for (could be a keyword or an ASIN (if it is ItemLookup i.e search by ID) and the Timestamp, Signature (base 64 Hmac) and of course the tags. Now we need to implement this in a real Java program. But the SOAP nature of the API could obviously cause some inconveniences.

Thankfully, Amazon made up a REST API code snippet which people can directly use. It takes in the URL as mentioned above, generates the timestamp, and signs the query with the Access ID, Associate Tag and the other params in the Hmac algorithm (which uses Base64). Here is the code: (SignedRequestsHelper.java)


/**********************************************************************************************
 * Copyright 2009 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file 
 * except in compliance with the License. A copy of the License is located at
 *
 *       http://aws.amazon.com/apache2.0/
 *
 * or in the "LICENSE.txt" file accompanying this file. This file is distributed on an "AS IS"
 * BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 * License for the specific language governing permissions and limitations under the License. 
 *
 * ********************************************************************************************
 *
 *  Amazon Product Advertising API
 *  Signed Requests Sample Code
 *
 *  API Version: 2009-03-31
 *
 */

package org.loklak.api.amazon;

import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Base64;
import java.util.Calendar;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.SortedMap;
import java.util.TimeZone;
import java.util.TreeMap;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

/**
 * This class contains all the logic for signing requests to the Amazon Product
 * Advertising API.
 */
public class SignedRequestsHelper {
	/**
	 * All strings are handled as UTF-8
	 */
	private static final String UTF8_CHARSET = "UTF-8";

	/**
	 * The HMAC algorithm required by Amazon
	 */
	private static final String HMAC_SHA256_ALGORITHM = "HmacSHA256";

	/**
	 * This is the URI for the service, don't change unless you really know what
	 * you're doing.
	 */
	private static final String REQUEST_URI = "/onca/xml";

	/**
	 * The sample uses HTTP GET to fetch the response. If you changed the sample
	 * to use HTTP POST instead, change the value below to POST.
	 */
	private static final String REQUEST_METHOD = "GET";

	private String endpoint = null;
	private String awsAccessKeyId = null;
	private String awsSecretKey = null;
	private String associatetag = null;
	private SecretKeySpec secretKeySpec = null;
	private Mac mac = null;

	/**
	 * You must provide the three values below to initialize the helper.
	 * 
	 * @param endpoint
	 *            Destination for the requests.
	 * @param awsAccessKeyId
	 *            Your AWS Access Key ID
	 * @param awsSecretKey
	 *            Your AWS Secret Key
	 */
	public static SignedRequestsHelper getInstance(String endpoint, String awsAccessKeyId, String awsSecretKey,
			String associatetag) throws IllegalArgumentException, UnsupportedEncodingException,
			NoSuchAlgorithmException, InvalidKeyException {
		if (null == endpoint || endpoint.length() == 0) {
			throw new IllegalArgumentException("endpoint is null or empty");
		}
		if (null == awsAccessKeyId || awsAccessKeyId.length() == 0) {
			throw new IllegalArgumentException("awsAccessKeyId is null or empty");
		}
		if (null == awsSecretKey || awsSecretKey.length() == 0) {
			throw new IllegalArgumentException("awsSecretKey is null or empty");
		}

		if (null == associatetag || associatetag.length() == 0) {
			throw new IllegalArgumentException("associatetag is null or empty");
		}

		SignedRequestsHelper instance = new SignedRequestsHelper();
		instance.endpoint = endpoint.toLowerCase();
		instance.awsAccessKeyId = awsAccessKeyId;
		instance.awsSecretKey = awsSecretKey;
		instance.associatetag = associatetag;

		byte[] secretyKeyBytes = instance.awsSecretKey.getBytes(UTF8_CHARSET);
		instance.secretKeySpec = new SecretKeySpec(secretyKeyBytes, HMAC_SHA256_ALGORITHM);
		instance.mac = Mac.getInstance(HMAC_SHA256_ALGORITHM);
		instance.mac.init(instance.secretKeySpec);

		return instance;
	}

	/**
	 * The construct is private since we'd rather use getInstance()
	 */
	private SignedRequestsHelper() {
	}

	/**
	 * This method signs requests in hashmap form. It returns a URL that should
	 * be used to fetch the response. The URL returned should not be modified in
	 * any way, doing so will invalidate the signature and Amazon will reject
	 * the request.
	 */
	public String sign(Map params) {
		// Let's add the AWSAccessKeyId, AssociateTag and Timestamp parameters
		// to the request.
		params.put("AWSAccessKeyId", this.awsAccessKeyId);
		params.put("AssociateTag", this.associatetag);
		params.put("Timestamp", this.timestamp());

		// The parameters need to be processed in lexicographical order, so
		// we'll
		// use a TreeMap implementation for that.
		SortedMap sortedParamMap = new TreeMap(params);

		// get the canonical form the query string
		String canonicalQS = this.canonicalize(sortedParamMap);

		// create the string upon which the signature is calculated
		String toSign = REQUEST_METHOD + "\n" + this.endpoint + "\n" + REQUEST_URI + "\n" + canonicalQS;

		// get the signature
		String hmac = this.hmac(toSign);
		String sig = this.percentEncodeRfc3986(hmac);

		// construct the URL
		String url = "http://" + this.endpoint + REQUEST_URI + "?" + canonicalQS + "&Signature=" + sig;

		return url;
	}

	/**
	 * This method signs requests in query-string form. It returns a URL that
	 * should be used to fetch the response. The URL returned should not be
	 * modified in any way, doing so will invalidate the signature and Amazon
	 * will reject the request.
	 */
	public String sign(String queryString) {
		// let's break the query string into it's constituent name-value pairs
		Map params = this.createParameterMap(queryString);

		// then we can sign the request as before
		return this.sign(params);
	}

	/**
	 * Compute the HMAC.
	 * 
	 * @param stringToSign
	 *            String to compute the HMAC over.
	 * @return base64-encoded hmac value.
	 */
	private String hmac(String stringToSign) {
		String signature = null;
		byte[] data;
		byte[] rawHmac;
		try {
			data = stringToSign.getBytes(UTF8_CHARSET);
			rawHmac = mac.doFinal(data);
			signature = Base64.getEncoder().encodeToString(rawHmac);
		} catch (UnsupportedEncodingException e) {
			throw new RuntimeException(UTF8_CHARSET + " is unsupported!", e);
		}
		return signature;
	}

	/**
	 * Generate a ISO-8601 format timestamp as required by Amazon.
	 * 
	 * @return ISO-8601 format timestamp.
	 */
	private String timestamp() {
		String timestamp = null;
		Calendar cal = Calendar.getInstance();
		DateFormat dfm = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");
		dfm.setTimeZone(TimeZone.getTimeZone("GMT"));
		timestamp = dfm.format(cal.getTime());
		return timestamp;
	}

	/**
	 * Canonicalize the query string as required by Amazon.
	 * 
	 * @param sortedParamMap
	 *            Parameter name-value pairs in lexicographical order.
	 * @return Canonical form of query string.
	 */
	private String canonicalize(SortedMap sortedParamMap) {
		if (sortedParamMap.isEmpty()) {
			return "";
		}

		StringBuffer buffer = new StringBuffer();
		Iterator<Map.Entry> iter = sortedParamMap.entrySet().iterator();

		while (iter.hasNext()) {
			Map.Entry kvpair = iter.next();
			buffer.append(percentEncodeRfc3986(kvpair.getKey()));
			buffer.append("=");
			buffer.append(percentEncodeRfc3986(kvpair.getValue()));
			if (iter.hasNext()) {
				buffer.append("&");
			}
		}
		String cannoical = buffer.toString();
		return cannoical;
	}

	/**
	 * Percent-encode values according the RFC 3986. The built-in Java
	 * URLEncoder does not encode according to the RFC, so we make the extra
	 * replacements.
	 * 
	 * @param s
	 *            decoded string
	 * @return encoded string per RFC 3986
	 */
	private String percentEncodeRfc3986(String s) {
		String out;
		try {
			out = URLEncoder.encode(s, UTF8_CHARSET).replace("+", "%20").replace("*", "%2A").replace("%7E", "~");
		} catch (UnsupportedEncodingException e) {
			out = s;
		}
		return out;
	}

	/**
	 * Takes a query string, separates the constituent name-value pairs and
	 * stores them in a hashmap.
	 * 
	 * @param queryString
	 * @return
	 */
	private Map createParameterMap(String queryString) {
		Map map = new HashMap();
		String[] pairs = queryString.split("&");

		for (String pair : pairs) {
			if (pair.length() < 1) {
				continue;
			}

			String[] tokens = pair.split("=", 2);
			for (int j = 0; j < tokens.length; j++) {
				try {
					tokens[j] = URLDecoder.decode(tokens[j], UTF8_CHARSET);
				} catch (UnsupportedEncodingException e) {
				}
			}
			switch (tokens.length) {
			case 1: {
				if (pair.charAt(0) == '=') {
					map.put("", tokens[0]);
				} else {
					map.put(tokens[0], "");
				}
				break;
			}
			case 2: {
				map.put(tokens[0], tokens[1]);
				break;
			}
			default: {
				// nothing
				break;
			}
			}
		}
		return map;
	}
}

Now things become a whole lot easier. We can straightaway sign our requests using this class, make our request authenticated, and get the result.

Now we need to figure out what we should get from the API. My idea was to use the Large ResponseGroup by default, so that we get all the possible info (the Large ResponseGroup encapsulates all other ResponseGroups), and also, we should enable searching both by ASIN and Product Name so that the API is efficient enough and can give proper results, i.e I had to implement both the ItemLookup and ItemSearch APIs. Also, I added an option to choose your own ResponseGroup so that you can select what all quantity of data, and what all data you want, and get the result.

So here is the code of the AmazonAPIService, which enables Susi Monetisation.


/**
 *  AmazonProductService
 *  Copyright 05.08.2016 by Shiven Mian, @shivenmian
 *
 *  This library is free software; you can redistribute it and/or
 *  modify it under the terms of the GNU Lesser General Public
 *  License as published by the Free Software Foundation; either
 *  version 2.1 of the License, or (at your option) any later version.
 *  
 *  This library is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 *  Lesser General Public License for more details.
 *  
 *  You should have received a copy of the GNU Lesser General Public License
 *  along with this program in the file lgpl21.txt
 *  If not, see .
 */

package org.loklak.api.amazon;

import java.io.StringWriter;

import javax.servlet.http.HttpServletResponse;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;

import org.json.JSONObject;
import org.json.XML;
import org.loklak.data.DAO;
import org.loklak.server.APIException;
import org.loklak.server.APIHandler;
import org.loklak.server.AbstractAPIHandler;
import org.loklak.server.Authorization;
import org.loklak.server.BaseUserRole;
import org.loklak.server.Query;
import org.loklak.tools.storage.JSONObjectWithDefault;
import org.w3c.dom.Document;

public class AmazonProductService extends AbstractAPIHandler implements APIHandler {

	private static final long serialVersionUID = 2279773523424505716L;

	// set your key configuration in config.properties under the Amazon API
	// Settings field
	private static final String AWS_ACCESS_KEY_ID = DAO.getConfig("aws_access_key_id", "randomxyz");
	private static final String AWS_SECRET_KEY = DAO.getConfig("aws_secret_key", "randomxyz");
	private static final String ASSOCIATE_TAG = DAO.getConfig("aws_associate_tag", "randomxyz");

	// using the USA locale
	private static final String ENDPOINT = "webservices.amazon.com";

	@Override
	public String getAPIPath() {
		return "/cms/amazonservice.json";
	}

	@Override
	public BaseUserRole getMinimalBaseUserRole() {
		return BaseUserRole.ANONYMOUS;
	}

	@Override
	public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
		return null;
	}

	public static JSONObject fetchResults(String requestUrl, String operation) {
		JSONObject itemlookup = new JSONObject(true);
		try {
			DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
			DocumentBuilder db = dbf.newDocumentBuilder();
			Document doc = db.parse(requestUrl);
			DOMSource domSource = new DOMSource(doc);
			StringWriter writer = new StringWriter();
			StreamResult result = new StreamResult(writer);
			TransformerFactory tf = TransformerFactory.newInstance();
			Transformer transformer = tf.newTransformer();
			transformer.transform(domSource, result);
			JSONObject xmlresult = new JSONObject(true);
			xmlresult = XML.toJSONObject(writer.toString());
			JSONObject items = xmlresult.getJSONObject(operation).getJSONObject("Items");
			if (items.getJSONObject("Request").has("Errors")) {
				itemlookup.put("status", "error");
				itemlookup.put("reason",
						items.getJSONObject("Request").getJSONObject("Errors").getJSONObject("Error").get("Message"));
				return itemlookup;
			}
			itemlookup.put("number_of_items",
					(operation.equals("ItemLookupResponse") ? "1" : (items.getJSONArray("Item").length())));
			itemlookup.put("list_of_items", items);
		} catch (Exception e) {
			itemlookup.put("status", "error");
			itemlookup.put("reason", e);
			return itemlookup;
		}
		return itemlookup;
	}

	@Override
	public JSONObject serviceImpl(Query call, HttpServletResponse response, Authorization rights,
			JSONObjectWithDefault permissions) throws APIException {
		String ITEM_ID = call.get("id", "");
		String PRODUCT_NAME = call.get("q", "");
		String responsegroup = (call.get("response_group", "") != "" ? call.get("response_group", "") : "Large");
		if (!("".equals(ITEM_ID)) && ITEM_ID.length() != 0) {
			return itemLookup(ITEM_ID, responsegroup);
		} else if (!("".equals(PRODUCT_NAME)) && PRODUCT_NAME.length() != 0) {
			return itemSearch(PRODUCT_NAME, responsegroup);
		} else {
			return new JSONObject().put("error", "no parameters given");
		}
	}

	public JSONObject itemSearch(String query, String responsegroup) {
		JSONObject result = new JSONObject(true);
		SignedRequestsHelper helper;
		if (query.length() == 0 || "".equals(query)) {
			result.put("error", "Please specify a query to search");
			return result;
		}
		try {
			helper = SignedRequestsHelper.getInstance(ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_KEY, ASSOCIATE_TAG);
		} catch (Exception e) {
			result.put("error", e.toString());
			return result;
		}
		String requestUrl = null;
		String queryString = "Service=AWSECommerceService&ResponseGroup=" + responsegroup
				+ "&Operation=ItemSearch&Keywords=" + query + "&SearchIndex=All";
		requestUrl = helper.sign(queryString);
		result = fetchResults(requestUrl, "ItemSearchResponse");
		return result;
	}

	public JSONObject itemLookup(String asin, String responsegroup) {
		SignedRequestsHelper helper;
		JSONObject result = new JSONObject(true);
		if (asin.length() == 0 || "".equals(asin)) {
			result.put("error", "Please specify an Item ID");
			return result;
		}

		try {
			helper = SignedRequestsHelper.getInstance(ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_KEY, ASSOCIATE_TAG);
		} catch (Exception e) {
			result.put("error", e.toString());
			return result;
		}
		String requestUrl = null;
		String queryString = "Service=AWSECommerceService&ResponseGroup=" + responsegroup
				+ "&Operation=ItemLookup&ItemId=" + asin;
		requestUrl = helper.sign(queryString);
		result = fetchResults(requestUrl, "ItemLookupResponse");
		return result;
	}

}

As you can see in this code, I have taken in the parameters (either of q or ASIN, and responsegroup), and depending on type of param, I have decided whether to use the ItemLookup or the ItemSearch API (only these two as of now are relevant for Susi in real). The ResponseGroup is defaulted to Large, so even if you avoid the responsegroup param, you still get all the data. What next? I just built the query, signed it using the SignedRequestsHelper (note: the associate tags and the keys are in the config file as mentioned in my last blog post), and I then parse the returned XML and display it as a JSON.

We are yet to get this into Susi (in the form of questions), but that will be up soon. Susi can simply be monetised by sending in the URL (which contains our associate tag) along with the result, so that a person can go to the URL and we can get hits on that, for which we get paid by the Affiliates Program. But now, we have seen how we intend the API to work. Since the Product Advertising API is huge, we can always make this API more efficient and expand it, which is a future plan too.

Feedback, as always, is welcome. 🙂

Monetisation of Susi using the Amazon Product API (Part 2)