The share of a reproduction paper is based on analyses of as well as in insights into current methods and problems—plus the additional certainty that is sold with validating past outcomes.

1. Relevance: Is it paper highly relevant to COLING?
2. Readability/clarity: could be the paper well-structured and well-written?
3. Data/code supply: may be the data/code (as appropriate) open to the study community or is here a compelling explanation provided why this isn’t feasible?
4. Analysis: If the paper managed to reproduce the total outcomes of the previous work, does it demonstrably set down exactly exactly what would have to be filled in to carry out therefore? If it wasn’t in a position to reproduce the outcomes of earlier in the day work, does it plainly identify exactly what information was missing/the most likely causes?
5. Generalizability: Does the paper exceed replicating the outcomes in the initial to explore if they could be reproduced an additional environment? Instead, in situations of non-replicability, does the paper discuss the wider implications of the outcome?
6. Informativeness: To exactly exactly what level does the analysis reported into the paper deepen our comprehension of the methodology utilized or perhaps the nagging issue approached? Will the information within the paper assistance professionals along with their range of technique/resource?
7. Significant contrast: as well as determining the experimental outcomes being replicated, does the paper motivate why these specific email address details are a crucial target for reproduction and exactly what the near future implications are of these having been reproduced or been discovered to be non-reproducible?
8. General suggestion: there are numerous good submissions contending for slots at COLING 2018; essential will it be to feature that one? Will people discover great deal by looking over this paper or seeing it presented? Please be decisive—it is much better to vary from other reviewers rather than grade every thing in the middle.

Site paper

Documents in this track provide a language resource that is new. This might be a corpus, but additionally might be an annotation standard, device, an such like.

1. Relevance: Is this paper highly relevant to COLING? Will the resource presented likely be of good use to the community?
2. Readability/clarity: From the real means the paper is created, are you able to inform how a resource ended up being produced, the way the quality of annotations (if any) ended up being assessed, and exactly why the resource must certanly be of great interest?
3. Originality: Does the resource fill a need into the current number of available resources? Observe that originality could be when you look at the range of language/language genre or variety, within the design associated with the annotation scheme, when you look at the scale associated with the resource, or nevertheless other parameters.
4. Site quality: what type of quality control had been performed? If appropriate, had been inter-annotator contract calculated, and in case therefore, with appropriate metrics? Otherwise, how many other assessment ended up being carried out, and exactly how agreeable were the outcome?
5. Site accessibility: might it be simple for researchers to down load or access that is otherwise resource to be able to utilize it in their own personal work? As to the level could work centered on this resource be provided? answers to add: Yes, i’ve verified
6. Metadata: perform some writers explain whoever language use is captured within the resource also to just just what populations results that are experimental regarding the resource might be generalized to? In the event of annotated resources, will be the demographics associated with the annotators also characterized?
7. Meaningful comparison: Is the brand new resource situated with regards to current operate in the industry, including comparable resources it took motivation from or improves on? Can it be clear what exactly is unique about the resource?
8. General suggestion: there are lots of submissions that are good for slots at COLING 2018; essential could it be to feature that one? Will people discover a complete great deal by looking over this paper or seeing it presented? Please be decisive—it is way better to change from other reviewers rather than grade every thing in the centre.

Position paper

A situation paper presents a challenge to main-stream reasoning or a futuristic brand new eyesight. It might open an area that is new unique technology, propose changes in existing research, or offer a brand new pair of ground guidelines.

1. Relevance: Is it paper highly relevant to COLING?
2. Readability/clarity: Is it clear just exactly what the positioning is the fact that paper is arguing for? Would be the arguments because of it laid call at a way that is understandable?
3. Soundness: Are the arguments presented within the paper appropriate and coherent? May be the eyesight well-defined, with success requirements? (Note: it must be feasible to offer a score that is high even although you don’t concur with the place taken because of the writers)
4. Imagination: How unique or bold may be the place drawn in the paper? Does it express well-thought through and innovative ground that is new?
5. Range: How much range for brand brand new scientific studies are opened by this paper? just What effect could it have on current areas and concerns?
6. Meaningful contrast: could be the paper well-situated with regards to past work, both place documents (taking the exact same or opposing side on a single or comparable dilemmas) and appropriate theoretical or work that is experimental?
7. Substance: Does the paper have sufficient substance for a full-length paper? Could be the presssing problem adequately essential? Will be the arguments adequately varied and thoughtful?
8. General suggestion: there are lots of submissions that are good for slots at COLING 2018; essential can it be to feature that one? Please be decisive—it is much better to change from other reviewers rather than grade every thing in the center.

A study paper provides an organized summary of the literary works up to now for a particular subject that assists the reader understand the kinds of concerns being inquired about the subject, the many approaches which were used, the way they relate genuinely to one another, and just exactly exactly what further research areas they start. A conference-length study paper social psychology research topic should sufficiently be about a concentrated subject so it can try this effectively with into the web web page limits.

1. Relevance: may be the paper highly relevant to COLING?
2. Readability/clarity: could be the paper generally speaking very easy to follow and well organized?
3. Organization: Does the paper arrange the appropriate literary works in a narrative and determine typical strands of inquiry?
4. Scope: Does the paper determine a fairly concentrated area to survey?
5. Thoroughness: provided the region identified to survey, does the paper cover most of the literature that is relevant? May be the literary works evaluated represented accurately?
6. Outlook: Does the paper recognize areas for future work and/or point out what clearly just isn’t yet managed inside the literary works surveyed?
7. Context: Does the paper situate research that is current within its historic context? (We don’t expect papers to begin with Pa?ini, yet in the exact same time one thing that just cites work from 2017 most likely does not capture exactly how current work pertains to the larger image.)