Free Mobile & Web Survey Application for easy data collection using Open Data Kit (ODK)

After trying and writing post about EpiCollect5 I decided to look at another data collection tool, Open Data Kit (ODK). A brief overview of it in video below:

This has some strengths and weaknesses over EpiCollect5.

Weaknesses:

1. Only Android App, not Apple (but since there are lots of cheap Android mobiles this is not a major problem)
2. Cannot update data in a partially completed form very easily
3. Cannot display GPS raw capture data easily on Aggregate Map display easily without altering the GPS android capture of Latitude & Longitude in 2 separate columns (you need to combine them into a joint column)

Strengths. Quite a few in my opinion:

1. You can setup your own free server (to run Aggregate programme) to run it on, (in some ways a pain as you have to set it up, but it is easy to setup with a google cloud account) and you have total control over the data. You are not posting data to someone else’s website/cloud.
2. The XML form structure can be:
   1. Developed and saved in a web-browser using  ODK Build and you can save the form as a .odkbuild file type
   2. You can download ODK Build here and run it offline on your PC and you can save the form as a .odkbuild file type 
   3. You can use Excel to create XLSForms (based on 2 tabs on a worksheet) that you can then convert to an XMLForm using a converter web page here.
   4. The XLForms is an open standard that can be used in a number of free and paid Mobile Survey programmes. See here for form structure and rules. Here is a link to the Open Data Kit page for Form logic.
   5. There are other programmes that you can use to design your form, some free, some paid. These work with the free ODK android app that you can get from Android Play Store
   6. Unlike the EpiCollect5 programme, you can have groups nested within groups, and instead of children you can create loops (eg for rooms, if you want to keep on adding a room, just use a loop (instead of a child in EpiCollect5). So the Form structure is more versatile.
   7. You can edit/embellish your forms and do URL links. See here.
3. The programme seems to be more versatile than EpiCollect5 in that a lot of other paid and free programmes can use the main form structure.
4. You can collect Metadata automatically so that surveyor does not need to fill some of the information in:XLSForm has a number of data type options available for meta data collection:
   

   Metadata type	Meaning
   start	Start date and time of the survey.
   end	End date and time of the survey.
   today	Day of the survey.
   deviceid	IMEI (International Mobile Equipment Identity)
   subscriberid	IMSI (International Mobile Subscriber Identity)
   simserial	SIM serial number.
   phonenumber	Phone number (if available).
   username	Username configured (if available).
   email	Email address configured (if available).

   Note that some metadata fields only apply for mobile phone-based forms.

5. External XML data. For advanced users, who need to perform complex queries on external data without restrictions, an external XML data file can be added with question type xml-external. The value in the name column can be used to refer to this data in any formula (e.g. for a calculation, constraint, relevant, or choice_filter) using the instance(‘name’) function. A file with the same name and the .xml extension should be uploaded with the form. See below for an example that requires uploading a file called houses.xml with the form.

   survey
   	type	name	label	calculation
   	xml-external	houses
   	integer	rooms	How many rooms?
   	calculate	count		count(instance(‘houses’)/house[rooms = current()/../rooms ])

Other packages

As mentioned above, there are other packages that can develop forms for the ODK App.

There is also a further package, Geographical Open Data Kit, that focuses on geographical data and allows points and poly-lines to be marked on the survey. It uses a different app that can be found here. It uses the Aggregate programme and the server setup used for the ODK setup. I have not tested this programme yet. Possibly the issue with showing data on the Aggregate map mentioned above may be overcome with this process.

Aggregate, setup and use

There are several choices for setting up Aggregate programme. Go here for choices.

Although I have a VPS I would need to get another Virtual Server up and running for it, and I already have Tomcat running OpenMaint & Alfresco, so I thought for testing purposes I would use the Free Google App Engine (only free for a year with reasonable loading). The process is described here. You do have to give visa details to create an account.

I found the install process quite easy, but I got muddled with ID and name of the Google App Engine and kept on inputting the wrong one so had to loop back through until I read the instructions right. I ended up resorting to YouTube and found this video really helpful.

The rest of his videos are worth watching too.  You can find them here.

You will also need to look at the building of forms. YouTube has a few videos on that, although for some of the loop/group processes I ended up doing a general search on XLSForms to find some resources.

Like EpiCollect5 when you download survey data to CSV it exports it as a single CSV file, so if you have several tiers of Data , like 1/site Data/information, 2/Building Data such as exterior works, 3/Room data, walls, ceiling floor, door, hardware, fittings etc. On export you will need to have 3 CSV files, one for Buildings, one for External, one for Rooms.

Also, there is the parent/Child link with the parentID (Site) shown on the External CSV and the Room CSV, this links these items with the Site CSV.

I haven’t tried to join this data together yet. Either Excel for connection or Python. An exercise for another day.

My Process

I initially started using the ODK Build online but moved over to looking at XLSForms with excel.

The reason was that it was easier to edit, compile to XML (XForm) and email to the mobile phone for testing.  Doing the editing/programming in Excel you could do that off-line and keep the code. I was working on 2 computers, one with an internet connection all the time the other with intermittent connection, so found the off-line flow in Excel easier.

I can slowly build the code up to include extra choices such as wall , floor and ceiling types and could reference the Excel sheet with the SPM data easily and use copy/paste.

I found that you could do the drill down to different levels quite easily because you can nesta group within another group.

There is the opportunity to do linking to other URL’s and to do some styling, see here.

I haven’t tried these yet, but the link to URL could be useful as a link to extra information/documentation about the survey to assist the surveyor. I’m also interested if you can re-format the form to add more information on one screen. You do a lot of swiping (from right to left on the screen) as a method of moving onto the next question.

The other thing I haven’t tested (both in ODK and EpiCollect5) is using the apps on a tablet. I have tested them on a 5.5″ phone screen in portrait mode only.

I did try creating an account for KoboToolkit but it wouldn’t let me create an account. The interface for the form builder looks pretty good for that. See this video.

The XML format is a data format , so when editing the Excel pages you can colour/highlight/underline info to make the forms easier to see and this formatting will not be transferred to the XForm, but it makes it easier to read the code when developing the form.

Here is the link to the free ODK Collect android app on Google Play.

Here are some screenshots of the apps versatility.

Note, when getting GPS  on the app for location you can have a background map to show the location, but this uses extra resources and doesn’t work online, so best not to have map background unless you are sure of wifi/3/4g connection.

Uploading forms to ODK Aggregate Server

Using the XLSForm upload, select a file (1) and then Submit it (2). If there are errors it will tell you at the bottom (light tan/yellow area) you need to fix these before it will let you go any further.

Once you see the green buttons, you can use the right hand button to Preview the Form in Enketo (which is really useful (see imge below) andf use the right green button to download the XForm file (with .xml extension)

This is the Preview of the Form in Enketo

Aftewr downloading to your PC you can then open your Server ODK Aggregate.

Go to middle tab and select Add new Form and in the pop up box select the .xml file and upload it.

If you already have a version of that compiled xml file  with the same form_id in the settings tab, then it will say the form is already uploaded.

You have 2 choices:

delete the old one ( if you have no data uploaded on that form) or

give the new form a new version and form_id (on the settings tab in the XLS file)

From the App on the phone you can download a BLANK FORM from the server (on the Menu page of the APP).

You can directly upload to the app a compiled XML file but a good workflow is to use the Server as the source of the blank forms.

Two reasons for this:

1.  The server version should be the latest version, and if uploading onto lots of phones, then you want them all to be the same.
2.  The data will only be uploaded onto the server if the version of the form is the same.

 End Comment

This tool is a little more complex than EpiCollect5 and will certainly allow you do have a more versatile form for data collection. The fact that the information is uploaded to your server (in this case Google app Engine) means you have control of that data.

A couple of issues that I have found with ODK.

1. If you only partially fill out a form and upload it to the server, that form has gone from the mobile device and is now residing on the server only. To put it back on the device to complete it requires you to go to the server via a browser and download a CSV of that Survey and load it onto the mobile device that you are using to do the update of that data. I have not found an easy way of doing this. It also seems a bit awkward, as you will most probably go back and do a follow up of a survey and need the original data, it seems a bit crude. This would mean uploading about 3 CSV files for the nested survey that I was developing (process yet to be tested).
2. Displaying Locations of GPS data on the map on the server in the Web Browser in the Aggregate programme. When capturing the GPS Lat/Long data with the GPS it puts latitude in one field and longitude in another. So when you come to try and display the data on the map app in the Aggregate programme it asks for the location expecting a Lat,Long field (a field with latitude and longitude all in the same cell with a comma separator. It only gives you one column to select. So you have to make (I think) in the form a calculated field that will combine lat& long in a singe column cell. Seems a bit daft, an extra bit of effort for something that should have been incorporated programmatically into the programme.

I will certainly be exploring developing some more apps with this tool, also the Geographical ODK programme too.

In the Aggregatedownload to CSV file there is a link to the photos in one of the cells (if a photo has been taken in the survey). This link downloads the photo from the server. I do need to go back and check if this is the case with the EpiCollect5 photo access that I mentioned in the previous post.