PopUpSurvey
What (summary)
Short survey of one or two questions pop up for some percentage of our users. Questions will be varied by day and repeated by week.
This capability was most recently discussed in staff email April 25-28 under the subject: welcoming modal, survey. Vinh started this conversation by offering the mockup that appears to the right.
Why this is important
We are continually challenged to "Know our Customer" in a quantitative way. We need something simple and believable that gives us percentages and trends. This is input to our fund raising efforts.
DoneDone
Questions and question presentation parameters can be entered and modified by a list of specific users, initially "Ray King", "Julia", "Kasey", "Vinh". A simple rails controller, "SurveyController", will provide access to the question pool and online stats.
Statistics appear in real time, with a lag hopefully measured in minutes but not to exceed 24 hours. Statistics include how many times a question was asked per day and what percentage of answers were given, including no answer (how many ways?) There must be some reasonable path from these reports into Microsoft Excel.
It must be possible to offer the opportunity to enter open-ended responses. These would appear as a text box that could hold up to five lines. These will appear in the stats as a percentage of people entering a comment. A second report will show all such responses for a specific question on a given day.
It should allow us to present the survey to X% of users and stop after acquiring Y number of responses.
The design needs to be clear and compelling so that we achieve a reasonable response rate. (Can we find industry standards for expected response rates?)
Some record, cookie or db, needs to record who has been surveyed so as to not repeat questions.
Some automated routine is worked out to cycle questions without continuous operation. (deferred)
At launch there will be hypotheses to test and expected thresholds for interpreting response rates.
Steps to get to DoneDone
- Choose conditions under which we display the popup.
- User comes only from Google or other search engine?
- I assume search engines themselves should never see popup code. -- Stephen
-
Let's start by presenting the popup to 1% of all visitors to domain pages being careful to not present to the same ip address twice. Later we can make these parameters options.
-
Some sample questions (and hypotheses) worked up and added to this page. -- JuliaThe questions are here: https://admin.aboutus.com/index.php/CustomerSurvey
-
Mockup of question PopUp mechanism, independent of question database.-- Vinh -
Database design, migrations and skeleton screens made for rails.-- Lahore -
Refined hypotheses and corresponding questions for day 1 questions.-- Julia -
First integration of PopUp and Database, including basic statistics.-- Team -
Trial run for one day. Review results. Propose changes.-- Team and Ray -
Begin routine surveying with better questions and automatic progression.
additional requests:
- some way to substitute the page title into the response text, maybe gsub for example.com?
Database Design
I suggest simplifications to this design. See reputation below. -- Ward
Hello developers! Ali Aslam and I have just pushed the models that we think are necessary to store various information regarding survey questions, their answers, and statistics related to them. We think it will be good if you know of the design so far, and how it will be used. Initially we designed the survey and related tables thinking we would have non-adaptive surveys, but adding the infrastructure to make them dynamic has increased complexity to some extent, so we think an explanation is not out of place.
Question: This table has the question text, and the type of the question. The type of the question determines how this question would be displayed to the user. Types can be 1)MCQ, 2)MCQ with an open-ended question that leaves open the possibility that the user will enter a line or a paragraph in response, 3) single line response, 4) multiple line or text response. So far, we are not really considering that the user may be allowed to select multiple answers instead of just one from the mcq list
QuestionOption: if the question has mcqs, they go in here
Answer: User responses/selections are stored here
Survey: Each survey consists of a number of questions, and the next question to come depends on the answer to the previous question. Each entry stores the start question's id.
Next question: Each entry contains the id of the question to ask, and the option user opts for. these two pieces of information determine the id of the next node, that is, the id of the next "Next Question" entry. (This becomes a linked list as a result). So we keep fetching next questions based on the responses we get till we are done. There is a possibility of cycles, but we will get over it by defining max number of questions we can ask in a survey.
We need to resolve one issue. Suppose if a question has two possibilities: Agree/Disagree, and this question gets asked for a week. Then someone edits the question and changes the options to: Disagree/Agree, and... u can imagine what i am driving it! The easiest way to resolve this is to train the question editors to not change semantics once options are selected, create new questions if they require major editing, etc, but this is certainly one issue that needs careful attention.
Looking forward to your feedback and suggestions for improvement!
Administrative Interface
This interface has been designed to be constructed with the Rails skeleton generators.
index page
http://aboutus.org/au_web_services/popup_survey/index
This page will list all of the survey popups that have or will run. Items will show an id number, a Date, and a count of responses. The date is used for scheduling one day runs on the site (utc time). Change the date to run the same question again. The count of responses is for the currently assigned date.
Each item will have links to Show, Edit and Destroy. Show will show the popup questions and list all answers. Edit will allow modification of the popup parameters (see next section). Destroy will remove the one popup and any answers recorded for it.
The page will have a Create link that will make a new popup and edit it.
edit page
http://aboutus.org/au_web_services/popup_survey/edit/1
This page allows for the modification of popup surveys. Entry fields will allow any sysop to change the date of a popup. Eventually other parameters will be present but for now these will be hard-wired into the site.
This page also allows for the modification of questions used in the popup. Questions will be numbered q1, q2, and so forth, but not all questions should be used in a given popup. Each question is of a predetermined type. One determines the style of the survey by choosing which of each type to use (not leave blank).
Questions q1 and q3 are either/or questions. These questions offer two choices, single words, like Yes/No, True/False, More/Less and so on. The entry field for specifying an either/or question looks like this:
q1 [ ] t [ ] f [ ]
The recorded results for either/or questions will be t, f, or nil.
Questions q2 and q4 are multiple-choice questions. These questions offer up to five choices appearing as short phrases. The entry field for specifying a multiple-choice question looks like this:
q2 [ ]
a [ ]
b [ ]
c [ ]
d [ ]
e [ ]
[ ] present choices in random order
The recorded results for multiple choice questions will be one, two, three, four, five (cooresponding to a - e) or nil. A check-box offers to randomize presentation order on survey display. If selected, the encoded question will include the binding :order => :random. Other orderings might be possible at a later date.
Question q5 and q7 are short-answer questions. This question prompts the visitor to write a sentence or two. The entry field for specifying a short-answer question looks like this:
q5 [ ]
The recorded results for a short-answer question will be the text entered as the short answer, or nil.
Question q6 is the 11-choice net-promoter-score (NPS) format question. The entry field for specifying this would have 11 radio buttons, labeled on each end with strongly agree and strongly disagree.
q6 Would you recommend this service to your colleagues? ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) strongly agree strongly disagree
show page
http://aboutus.org/au_web_services/popup_survey/show/1
This page reports the results of a specific popup_survey. It consists of one section for each question in use. After each question, the results for that question are tallied for the current day. (older data is tallied later)
For q1 and q3 the tally reports the number each of t, f, and nil.
For q2 and q4 the tally reports the number each of a, b, c, d, e, and nil.
For q5 and q7 the tally reports the number of each non-blank entry.
For q6 the tally reports the number of each of the 11 possible choices.
example
Let's say that we wanted to survey people to see if they are having fun on May 26th. We would go to the survey index page and press Create, enter the date of the survey, and then start specifying questions. We start with a true-false question:
q1 [Are you having fun yet? ] t [Yes ] f [No ]
We decide to follow this up with the more open-ended question, what would be more fun? This is a short-answer question so we skip to q5 to enter it:
q5 [What would be more fun? ]
We save this page and we are done. On May 26th we return to the index page and see that our survey has dozens of responses already. We click Show to see each question followed by the responses so far:
q1: Are you having fun yet? 20 Yes, 3 No, 1 nil
q5: What would be more fun? 18 nil 3 Playing Cricket 1 Twitter 1 Watching TV 1 Ward's Birthday Party
You will notice that this interface reports stats for only one date, the date as currently set in the survey. The Show page will offer to download stats for every day that the question ran. This will produce a CSV file with rows for dates and columns for response counts. Question q5, when used, will have tallies for nil and non-nil response only. (See au_action_controller for example CSV output.)
representation
The above survey could be recorded in a surveys table with fields:
id: 1
date: May 26, 2008
questions: {"q1": {"q": "Are you having fun yet?", "t": "Yes", "f": "No"},
"q5": {"q": "What would be more fun?"}}
The use of json for representing questions gives us flexibility that we are not yet using. For example, q6 could be a whole new kind of question with parameters yet to be considered.
Responses to the survey could be recorded in a survey_responses table with fields:
id: 24
survey_id: 1
ip: 245.132.61.19
date: May 26, 2008
answers: {"q1": "t", "q5": "Playing Cricket"}
We keep the ip address to avoid surveying the same place twice. Again json allows for flexibility for yet to be considered kinds of questions. The creation, presentation, recording and tallying methods should be extended in a single coordinated deployment.
This administrative interface and corresponding representation is sufficient to implement the originally proposed modal (see mockup above). It falls short of conditional questions (ask y in response to x). I will try to rewrite Julia's initial questions to fit within these constraints.
References
- We've compared our design to that recommended for web surveys and find that we have made satisfactory choices in most cases. Here are some of the pages that we've examined. Stephen Ginns confirms our desire to randomize choice order where appropriate.
- Ward built a much more complicated survey system for the Eclipse Foundation. One feature of this system was that everyone could post questions and everyone saw the results as they appeared. One hope was that moods would shift and opinions would refine through the week that each question was displayed. Folks could change their answer anytime. The question changed each Monday, and remained throughout the week.
