Coding Workflow
Follow these 10 steps every time you sit down to code a batch of comments
The value of this process depends entirely on your scores being genuinely independent. Do not look at, discuss, or compare scores with other coders until every coder has finished (Step 7).
Read the initiating post once
NewAt the start of the batch, read the initiating post — the report, clip or thread the comments respond to — together with the batch metadata. Without it, irony and shorthand in the comments are unreadable. Then set it aside: it is never scored.
Open only your own coder tab
Work only in your own sheet. Do not look at, discuss, or compare scores with other coders until every coder has finished (Step 7).
Check the pre-populated metadata
Confirm the Source Type (mainstream media or influencer), that the initiating-post identifier matches the batch, and that the parent comment is populated where this is a reply. A mis-recorded Source Type corrupts the comparison the whole study rests on.
Read the comment — and its parent
NewRead the target comment in full. Where it is a reply, read the comment it replies to, to identify who is being addressed — a reply's target is a fellow citizen. Everything you score must be present in the comment itself, not the thread around it.
Scan the Skip toggles
For each dimension, ask: is there any content here that could be scored on it at all? Use SKIP only when a dimension is structurally inapplicable — never because the comment is short or scores badly.
Score all indicators, add confidence
Score each indicator 0–4 against the absolute anchors — as if this were the only comment you had read that day. After each dimension, record one Confidence rating and use the notes field for sarcasm, slang or translation doubts.
Wait until all three have finished
Independence is not a suggestion. A single conversation between two coders before both have finished contaminates that comment's data.
Adjudicate flags as a team
Open the Reliability tab together. Look for Discuss flags (differences > 1 point), talk them through in the open, and agree a final score. Disagreement is data about the rubric, not a problem to tidy away.
Check the reliability statistics
Your target is 70% exact agreement or higher (Krippendorff's α ≥ 0.70). If below, review your coding before submitting. Every low-reliability indicator you identify is one ReHuman will also struggle with.
Save and submit the batch
Save the completed batch as instructed by your supervisor. Adjudicated batches become the ground truth ReHuman is trained and validated against.
The Confidence Column
After entering scores for all five indicators in a dimension, rate your overall confidence:
High Confidence
You are certain about all five scores. The comment was clear and the indicators applied straightforwardly.
Medium Confidence
You are fairly sure, but one or two indicators were ambiguous. You made a judgement call.
Low Confidence
The comment was genuinely difficult to score. Multiple indicators were ambiguous, or the comment was very short, heavily reliant on sarcasm or slang, or culturally unclear.
Do not default to Medium out of habit. Be honest about uncertainty. Low confidence scores are not a failure — they are important data.
The Skip Function — When a Dimension Doesn't Apply
SKIP means: there is simply no content in this comment that allows meaningful scoring on this dimension.
It does NOT mean the comment scores poorly on this dimension.
A comment that only says 'Happy birthday, mate!' under a report has no political content. You might skip all four dimensions.
A comment that has no empathy at all still gets a score of 0 on the Empathy indicators. Do not skip just because the comment scores badly.
Your target is 70% exact agreement or higher (Krippendorff's α ≥ 0.70). This is the standard threshold used across social science research for content analysis.