Guide · Career
A paper is not written to be read front to back by someone outside the field. It is written to survive review by four specialists. Once you know which sections are load-bearing and which are ceremony, most papers become readable in twenty minutes.
Papers are hard to read for a specific and non-obvious reason: the audience is not a curious outsider, it is a reviewer deciding whether to reject. Every section is optimised for that. The introduction exists partly to pre-empt "this has been done before." The methods section exists to answer "could this be replicated and did they do it properly." The related-work section is often as much about citing the right people as informing the reader.
That means reading front to back is close to the worst possible order. You spend the first fifteen minutes on the two sections least likely to change your mind about anything, and you arrive at the numbers already tired.
I have been on the writing side of this, as a co-author on a peer-reviewed paper accepted to SIGCSE TS 2026. What that experience mostly taught me is how much of a finished paper is shaped by anticipating objections, and how much genuinely useful information gets pushed into the two sections nobody reads: methods and limitations.
If you have five minutes and not twenty: abstract, figures, limitations. That triple tells you what was claimed, what was actually observed, and what the authors themselves say it does not prove. It is a startling amount of a paper for five minutes of work.
There is no universal threshold, which is why "n = 60, is that a lot?" has no answer in isolation. The right question is whether the sample is large enough to detect an effect of the size the authors are looking for, and whether it is representative of the population the claim is about.
| Question | What a good answer looks like | What should worry you |
|---|---|---|
| How many participants, and in each condition? | Stated plainly in the abstract or early methods, with the per-group split | Only a total, with the group sizes buried or absent |
| Was there a power analysis? | A stated target effect size and the sample needed to detect it, decided before data collection | No mention, especially alongside a barely-significant result |
| Who are the participants? | Described specifically: setting, background, recruitment method | "Undergraduates" with a claim generalised to "learners" |
| How many dropped out? | Reported, with a comparison of who left versus who stayed | A final n noticeably smaller than the recruited n, unexplained |
| How many outcomes were measured? | A small number, ideally pre-registered | Fifteen measures, one of which is significant and features in the title |
A small sample is not automatically bad. A carefully instrumented study of 20 people can be more informative than a sloppy survey of 5,000, and in fields like computing education the constraint is usually access to real classrooms rather than laziness. What a small sample does mean is that only large effects will show up at all, so a small study reporting a small significant effect deserves more scrutiny than a small study reporting a large one.
The subtler failure is representativeness. A sample drawn entirely from one university's introductory course is a fine basis for a claim about that population and a weak basis for a claim about students generally. Most overreach in papers happens at exactly this joint, and the limitations section usually admits it in one sentence that the abstract does not.
The formal definition is genuinely awkward, so here it is precisely and then in plain terms.
p = P(observing data at least this extreme | the null hypothesis is true) It is NOT: the probability the null hypothesis is true It is NOT: the probability the result was a fluke It is NOT: a measure of how big or important the effect isPlainly: a small p-value says "if there were genuinely no effect here, data this striking would be unusual." That is a statement about surprise under an assumption, not a statement about truth, and definitely not a statement about magnitude.
The 0.05 convention is a convention. It was never a law of nature, and treating it as a hard line produces two symmetric errors: dismissing p = 0.06 as nothing, and treating p = 0.04 as proof. Both readings ignore the number that actually matters, which is the effect size.
An effect size says how big the difference is, in units that survive comparison across studies. For a difference between two group means, the standard measure is Cohen's d.
Cohen's d = (mean_A - mean_B) / pooled standard deviation Rough conventions (and they are only conventions): d ≈ 0.2 small d ≈ 0.5 medium d ≈ 0.8 large Also common: r and R² for correlation, odds ratios for binary outcomes, and confidence intervals, which carry more information than any of them.The confidence interval deserves special attention because it does two jobs at once: it gives the estimate and it gives the uncertainty. An interval of [0.05, 0.90] and an interval of [0.44, 0.52] can have the same midpoint and mean entirely different things. And an interval that crosses zero is telling you the same thing a non-significant p-value would, but with the added and useful information of how large the effect could plausibly be in either direction.
The practical rule: read the effect size and the interval first, and let the p-value be a footnote. A significant result with a tiny effect and a wide interval is a weaker finding than a non-significant result with a moderate effect in a small sample, even though the first one is the one that gets a headline.
Concrete beats abstract, and it is easier to be honest about a paper with my own name on it. The SIGCSE TS 2026 paper I co-authored studied bilingual coding as a support for computer science learners, with 60 participants.
The paper reports statistically significant pre-to-post gains in programming confidence, computing identity, enjoyment, and motivation across those 60 participants, with novices gaining significantly more than experienced programmers. That is the real result and it is a good one.
Now the part that a casual reader would get wrong. The obvious headline someone would want from a bilingual coding study is "bilingual instruction beats English-only instruction." The paper does not support that. The bilingual-versus-English comparison produced a Cohen's d in the range of 0.25 to 0.40 and was not statistically significant. A d of 0.25 to 0.40 is a small-to-moderate effect in the direction you might hope for, and with this sample it is not distinguishable from no effect at all.
Two useful lessons come out of that, and they generalise well beyond this paper:
This is also a decent illustration of why exact numbers matter more than remembered impressions. A percentage figure attached to this study in casual retelling would be an invention, because the paper reports no percentage figures at all. If you cannot point to the number in the paper, do not repeat the number. The broader context of that work is in the bilingual coding guide, and the paper is described on the research page.
Most overreach is not fraud. It is a chain of small, individually defensible steps that ends somewhere the evidence never went. These are the joints where the chain usually slips.
| What the paper shows | What the discussion says | The gap |
|---|---|---|
| A correlation in observational data | "X improves Y" | Causal language on non-causal data. Look for random assignment; without it, be suspicious of any verb implying causation. |
| An effect in one specific population | "Students benefit from…" | Generalisation past the sample. Check who was actually enrolled. |
| A short-term outcome, measured immediately | "Improves learning" | Durability was never measured. Post-tests taken minutes after an intervention frequently do not survive a delayed re-test. |
| A self-reported survey measure | "Increases ability" | Confidence and competence are different constructs, and interventions routinely move one without the other. |
| One significant outcome among many | Headlined in the abstract | Multiple comparisons. Ask what else was measured and did not make the abstract. |
| A statistically significant tiny difference | "Substantially better" | Significance mistaken for magnitude. Go find the effect size. |
One quick heuristic that catches a surprising share of this: read the abstract's final sentence and the limitations section back to back. They are written by the same authors about the same study, and when they disagree in tone, the limitations section is the one telling the truth.
Having been through the process, the useful correction to most people's mental model is this: peer review is a check on reasoning and reporting, not a verification of the data. Reviewers do not re-run the study. They usually do not see the raw data at all.
| Peer review reliably catches | Peer review does not catch |
|---|---|
| Claims not supported by the reported results | Fabricated or mistaken data |
| Missing methodological detail that blocks replication | Whether the study replicates |
| Inappropriate statistical tests for the design | Analyses that were tried and abandoned before submission |
| Unacknowledged limitations and threats to validity | Whether the sample generalises in practice |
| Missing related work and mis-stated prior findings | Whether the effect matters in the real world |
So "peer reviewed" means a few knowledgeable people agreed the argument follows from the reported evidence and that the paper is worth the field's attention. It does not mean true. Replication, not review, is what does that job, and it happens on a much longer timescale than publication does.
Venue matters more than the phrase "peer reviewed" does. In computer science, top conferences are frequently the primary venue rather than journals, with acceptance rates and review processes that are public information. A preprint on arXiv has had no review at all, which is not disqualifying but is worth knowing before citing it. And any journal that solicits submissions by email with a fast fee-based turnaround should be treated as a marketing operation until proven otherwise.
That list takes a couple of minutes and it is most of what a reviewer does on a first pass. It will not make you a specialist in the field, and it does not need to. It is enough to tell a solid result from a stretched one, which is the entire practical skill. If you are heading in the other direction and want to be on the writing side, the undergraduate research guide covers how to get into a lab and what the first year actually looks like.
Multiple comparisons, effect sizes, and confidence intervals are not academic trivia. They are the same machinery that decides whether a trading strategy is real or a coincidence found by testing hundreds of parameter combinations against the same data. The failure mode is identical, and it is written up in exactly those terms in the backtest overfitting guide. If you understand why a paper that measured fifteen outcomes and reported one is weak evidence, you already understand why a strategy chosen from a thousand backtests is too.
FAQ
Abstract, then figures and their captions, then the limitations section, then methods, then results, then the discussion, and the introduction only if you need field context. Reading front to back spends your freshest attention on the two sections least likely to change your mind. If you only have five minutes, read the abstract, the figures, and the limitations.
It depends entirely on the effect being looked for and the population being claimed about. A small sample can only detect large effects, so a small study reporting a small significant result deserves more scrutiny than one reporting a large effect. Check whether a power analysis was done in advance and whether the participants represent the population in the conclusion.
It is the probability of observing data at least this extreme if the null hypothesis were true. It is not the probability that the hypothesis is true, not the probability the result was a fluke, and not a measure of how large the effect is. A small p-value says data this striking would be unusual if there were no effect, which is a statement about surprise, not about magnitude.
Significance answers whether an effect is distinguishable from zero; effect size answers how big it is. With a large enough sample a trivial difference becomes significant, and in a small sample a meaningful effect can fail to reach significance. Read the effect size and its confidence interval first and treat the p-value as a footnote.
No. It means this study could not distinguish the effect from zero, which is a statement about the study's resolving power as much as about reality. One of the papers I co-authored found a Cohen's d of roughly 0.25 to 0.40 on a bilingual-versus-English comparison that was not statistically significant, which is a small-to-moderate effect a larger sample might have resolved.
Read the abstract's last sentence and the limitations section back to back. The usual gaps are causal language on observational data, generalising past the sample that was actually studied, treating an immediate post-test as evidence of durable learning, and confusing a self-reported confidence measure with measured ability.
It checks reasoning and reporting, not data. Reviewers catch claims that outrun the results, missing methodological detail, inappropriate statistical tests, and unacknowledged limitations. They do not re-run the study, usually never see the raw data, and cannot tell you whether the finding replicates. Peer reviewed means the argument survived scrutiny, not that the conclusion is true.
Check arXiv and the author's own website first, since most venues permit an author-hosted version. Use an institutional login through a university library proxy for the ACM Digital Library, IEEE Xplore, and the major publishers. Look for an open-access flag. Failing all of that, email the author, which works far more often than people expect.