Why does go/token.FileSet give each file's EOF a position?

<p>See <a href="https://github.com/golang/go/blob/master/src/go/token/position.go#L404:" rel="nofollow">https://github.com/golang/go/blob/master/src/go/token/position.go#L404:</a></p> <pre><code>base += size + 1 // +1 because EOF also has a position </code></pre> <hr/>**评论：**<br/><br/>jerf: <pre><p>I doubt it&#39;s anything &#34;profound&#34;, it&#39;s probably just that they decided to A: represent EOF as a token, which is good for the parsing phase B: require that all tokens have a position and C: require that no two tokens overlap (which, if you think about it, also precludes tokens of size zero, since a size zero token would <em>always</em> overlap with its neighbors if you think about ranges of bytes). If you take those three properties of tokens, none of which are all that shocking on their own, all of which are sensible and are easier to deal with than trying to deal with tokens that violate those rules, then you end up implying that the EOF token must be one past the end of the file. (And presumably of length one, because why would you do it any other way? I mean, you <em>could</em>, but to what advantage?)</p></pre>indil7: <pre><p>I forgot there’s an EOF token. You’re probably right. Thanks!</p></pre>0xjnml: <pre><p>Because a N runes file has N + 1 scanner positions.</p> <p>Imagine a single letter file. In the beginning the scanner position is before the letter, after the file is scanned completely, the scanner is after the single letter (and at EOF). Those two positions are clearly distinct, aren&#39;t they?</p></pre>indil7: <pre><p>In that case, wouldn’t there need to be a position before the first letter too?</p> <p>Is that how the Go scanner actually works? I can imagine it instead working by starting at the letter, processing the first token, then advancing past the end of the file and stopping, no extra valid position required, it’s just past the end.</p></pre>0xjnml: <pre><blockquote> <p>In that case, wouldn’t there need to be a position before the first letter too?</p> </blockquote> <p>That&#39;s exactly how it is. In the one-letter example, two positions are needed as discussed before:</p> <blockquote> <p>In the beginning the scanner position is before the letter, after the file is scanned completely, the scanner is after the single letter (and at EOF).</p> </blockquote> <pre><code>| A | ----- ^ ^ | |__ after A, at EOF | |__ before A </code></pre> <p>Edit: formatting</p></pre>indil7: <pre><p>But there’s only an extra position after the end. There isn’t one before the beginning too.</p></pre>0xjnml: <pre><p>One letter, two positions. One before the letter, one after the letter. Are we looking at the same picture (above)?</p></pre>indil7: <pre><p>You’ve drawn three positions for a one-byte file.</p></pre>0xjnml: <pre><p>The picture consists of a single letter <code>A</code> and <em>two</em> position markers <code>|</code>. The first position marker is labeled <code>before A</code>, the second position marker is labeled <code>after A, at EOF</code>.</p> <p>There&#39;s no third position nor third position marker in the picture.</p></pre>apparentlymart: <pre><p>Along with what others already said, I can&#39;t speak for the Go scanner in particular but in my own adventures in scanning/parsing I always give EOF a position because then you have something to report in error messages that are triggered by an unexpected EOF, without having to special-case EOF with its own error message.</p> <p>That is, you can report e.g. &#34;unterminated string literal&#34; with the source position pointed at either a real token or EOF, rather than having to have a special &#34;unexpected EOF during string literal&#34; error case to pass this context on to the user.</p> <p>Additionally, if there are text editor build/verify steps integrated with your language then it gives the editor a specific location to show the squiggly red underline when an error occurs at EOF.</p></pre>indil7: <pre><p>What’s bad about simulating getting an unexpected EOF token by detecting there are no more tokens? You can still report the same error message if you want.</p> <p>Wouldn’t reporting an unexpected EOF token at a file offset outside the file be invalid for an editor?</p></pre>