Talking Web-Services to JIRA and Parsing JSON

In my previous post I mentioned a quick-n-simple way of parsing JSON web-service responses in C# with a helping hand from some other code. In this post I wanted to show a simple example of parsing the JIRA data. This is the basis of what I am using to do more more complex processing and then dumping to a simple CSV for more analysis in Excel.

Getting a single 'Issue'
At the basic level the following call can be used: https://jiraserver/rest/api/latest/issue/{0}
where the {0} is the number of the issue required. This effectively returns the same information that can be downloaded via already available web page functions.

However, in my case what I wanted to do was to work through the issue history and look at the state-transitions over a time period to analyse issue resolution. In this case the following call needs to be used: https://jiraserver/rest/api/latest/issue/{0}?expand=changelog

I've then got some pretty standard C# web-request code and reading the response back into a JSON string that I can then process:

static string GetIssue(string key)
{
    string url = String.Format("https://jiraserver/rest/api/latest/issue/{0}?expand=changelog",key);

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

    IWebProxy proxy = request.Proxy;
    if (proxy != null)
    {
        string proxyuri = proxy.GetProxy(request.RequestUri).ToString();
        request.UseDefaultCredentials = true;
        request.Proxy = new WebProxy(proxyuri, false);
        request.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;
    }


    string username = "username";
    string password = "password";
    string authInfo = username + ":" + password;
    authInfo = Convert.ToBase64String(Encoding.Default.GetBytes(authInfo));
    request.Headers["Authorization"] = "Basic " + authInfo;

    HttpWebResponse response = (HttpWebResponse)request.GetResponse();


    // we will read data via the response stream
    Stream s = response.GetResponseStream();

    // used on each read operation
    byte[] buf = new byte[8192];
    string tempString = null;
    int count = 0;

    // used to build entire input
    StringBuilder sb = new StringBuilder();

    do
    {
        // fill the buffer with data
        count = s.Read(buf, 0, buf.Length);

        // make sure we read some data
        if (count != 0)
        {
            // translate from bytes to ASCII text
            tempString = Encoding.ASCII.GetString(buf, 0, count);

            // continue building the string
            sb.Append(tempString);
        }
    }
    while (count > 0); // any more data to read?

    return sb.ToString(); // json
}

Processing the JSON response
The next step is to extract the data into an object that I can use for more processing. Keeping things simple this is a basic class containing just the data I needed. If I were using this more heavily I probably should do a proper job of JSON to object mappings (may for a follow-on post), but time here was really important, I knocked this up and completed the analysis in a couple of hours one afternoon and have been using it weekly ever since.

class Issue
{
    public string key;
    public string type;
    public string description;
    public string priority;
    public string component;
    public DateTime created;
    public string target;

    public class Transition
    {
        public DateTime date;
        public string from;
        public string to;

    }

    public List<Transition> transitions = new List<Transition>();
}

As processing the progression of issues was also a key point I added a simple hacky member to parse the date-time data:

public void SetCreated(string c)
{
    int yy = Convert.ToInt32(c.Substring(0, 4));
    int MM = Convert.ToInt32(c.Substring(5, 2));
    int dd = Convert.ToInt32(c.Substring(8, 2));
    int hh = Convert.ToInt32(c.Substring(11, 2));
    int mm = Convert.ToInt32(c.Substring(14, 2));
    int ss = Convert.ToInt32(c.Substring(17, 2));

    date = new DateTime(yy, MM, dd, hh, mm, ss);
}

I then had a simple function to process the JSON response into the class:

static Issue ProcessIssue(string json)
{
    JavaScriptSerializer jss = new JavaScriptSerializer();
    jss.RegisterConverters(new JavaScriptConverter[] { new DynamicJsonConverter() });
    DynamicJsonConverter.DynamicJsonObject j = jss.Deserialize(json, typeof(object)) as DynamicJsonConverter.DynamicJsonObject; //dynamic;

    dynamic d = jss.Deserialize(json, typeof(object)) as dynamic;

    Issue i = new Issue();

    i.key = d.key;
    i.type =  d.fields.issuetype.name;
    i.description = d.fields.summary;
    i.SetCreated(d.fields.created); 
    i.priority = d.fields.priority.name;

    Dictionary<string, object> fields = (Dictionary<string, object>)j._dictionary["fields"];
    if (fields.ContainsKey("customfield_11342") && fields["customfield_11342"] != null)
    {
        i.target = d.fields.customfield_11342.value;
    }


    foreach (dynamic h in d.changelog.histories)
    {
        foreach (dynamic item in h.items) // items (fields)
        {
            if (item.field == "status")
            {
                Issue.Transition t = new Issue.Transition();
                t.SetCreated(h.created);
                t.from = item.fromString;
                t.to = item.toString;

                i.transitions.Add(t);
            }
        }
    }

    return i;
}

This then gives enough for me to do the later processing I needed. You can see from this how easy the JSON parsing code is.

In testing I also put a simple routine together to enable easier checking of JSON and allow dumping of the data to XML in case I wanted to process further or use for other purposes. This also helped figure out some of the JSON structure. It's a bit hacky again, but does the job:

public static void MakeXML(XElement x, IDictionary<string, object> d)
{
    foreach (KeyValuePair<string, object> kv in d)
    {
        if (kv.Value is String)
        {
            x.Add(new XElement("field",
                    new XAttribute("key", kv.Key),
                    new XAttribute("value", kv.Value.ToString())));
        }
        if (kv.Value is Dictionary<string, object>)
        {
            XElement o = new XElement(kv.Key);
            x.Add(o);
            MakeXML(o, kv.Value as IDictionary<string, object>);
        }
        if (kv.Value is ArrayList)
        {
            XElement o = new XElement(kv.Key);
            x.Add(o);
            MakeXML(o, kv.Value as ArrayList);
        }
    }
}


public static void MakeXML(XElement x, ArrayList a)
{
    foreach (object o in a)
    {
        if (o is String)
        {
            x.Add(new XElement("field", o.ToString()));
        }
        if (o is Dictionary<string, object>)
        {
            XElement aa = new XElement("fields");
            x.Add(aa);
            MakeXML(aa, o as IDictionary<string, object>);
        }
        if (o is ArrayList)
        {
            XElement aa = new XElement("fields");
            x.Add(aa);
            MakeXML(aa, o as ArrayList);
        }
    }
}

This can be easily called in the earlier processing function as follows:

XElement x = new XElement("json");
MakeXML(x, j._dictionary as IDictionary<string, object>);

XDocument xdoc = new XDocument();
xdoc.Add(x);
xdoc.Save(String.Format("issue-{0}.xml",i.key));

Processing All Issues
Now that I can process a single issue I needed to do some analysis over all the issues, so another call is needed to get all the issues for a particular project:

https://jiraserver/rest/api/latest/search?jql=project={0}&maxResults=100&startAt={1}

As you can see this the project name is in the first field and the results are paged which adds some extra work. I'll omit the calling function as it's pretty much the same as the previous one for a single issue, it's the processing of the JSON that is important:

static List<string> ProcessIssues(string json)
{
    JavaScriptSerializer jss = new JavaScriptSerializer();
    jss.RegisterConverters(new JavaScriptConverter[] { new DynamicJsonConverter() });
    DynamicJsonConverter.DynamicJsonObject j = jss.Deserialize(json, typeof(object)) as DynamicJsonConverter.DynamicJsonObject; //dynamic;

    dynamic d = jss.Deserialize(json, typeof(object)) as dynamic;


    List<string> issues = new List<string>();

    foreach (dynamic i in d.issues)
    {
        issues.Add(i.key);
    }

    return issues;
}

Pretty basic code, in this case the response is to process through a list of the issues that come back and add them into a list that I can then use to call the individual issues later.

So, here's the code for calling the pages and getting all of the issues into a list:

string json;

int total = -1;
int index = 0;

List<string> allissues = new List<string>();

do
{
    json = GetIssues("PROJECT", index++);
    if (total < 0)
    {
        total = TotalIssues(json);
    }
    List<string> issues = ProcessIssues(json);

    foreach (string i in issues)
    {
        allissues.Add(i);
    }
}
while (index*100 < total);

It's not the most elegant bunch of code, but it's now a good tool that does a useful job for me. It took a couple of hours end-to-end one afternoon in the middle of other jobs and now saves considerably more time that the time it took to put together. Enjoy!