<p>No matter why useragent I use I get a 429 to many requests error from reddit</p>
<p>Here is my code </p>
<pre><code>package main
import (
"fmt"
"regexp"
"strings"
"log"
"net/http"
"os"
"bytes"
)
func stripchars(str, chr string) string {
return strings.Map(func(r rune) rune {
if strings.IndexRune(chr, r) < 0 {
return r
}
return -1
}, str)
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintf(os.Stderr, "Usage: %s URL\n", os.Args[0])
os.Exit(1)
}
re := regexp.MustCompile("https?://imgur.com/a......")
client := &http.Client{}
url := fmt.Sprintf("https://reddit.com/r/%s/top.json", os.Args[1])
resp, err := http.NewRequest("GET", url, nil)
resp.Header.Set("User-Agent", "linux:go-postgrabber:v0.1 (by /u/ineedmorealts)")
fmt.Printf("Getting" + url)
response, err := client.Do(resp)
if err != nil {
log.Fatal(err)
} else {
defer response.Body.Close()
buf := new(bytes.Buffer)
buf.ReadFrom(response.Body)
s := buf.String() // Does a complete copy of the bytes in the buffer.
links := re.FindString(stripchars(s, "\""))
fmt.Printf("%v\n", links)
fmt.Printf("%v\n",s)
}
}
</code></pre>
<p>Sorry if this isn't the right place for this</p>
<p>Edit: I have tired switching both useragent and IPs neither of which fixed the issue</p>
<p>Edit_2: <a href="/u/gohacker" rel="nofollow">u/gohacker</a> found the issue!</p>
<p><code>url := fmt.Sprintf("https://reddit.com/r/%s/top.json", os.Args[1])</code> should be <code>url := fmt.Sprintf("https://www.reddit.com/r/%s/top.json", os.Args[1])</code>, I have no idea why but this completely fixes the issue </p>
<hr/>**评论：**<br/><br/>afhtech: <pre><p>HTTP 429 is too many requests. Are you creating a new connection with every request instead of reusing just one? (New net/http instance on every request). On mobile but that's my guess.</p></pre>ineedmorealts: <pre><p>The program makes just make 1 request per run </p></pre>gohacker: <pre><blockquote>
<p><code>resp, err := http.NewRequest("GET", url, nil)</code></p>
</blockquote>
<p>Following <a href="https://github.com/Droogans/unmaintainable-code" rel="nofollow">the best practices</a>?</p>
<p>I fail to see how user agent is responsible for this. The only explanation that comes to mind is what <a href="/u/afhtech" rel="nofollow">u/afhtech</a> said. Are you invoking the program repeatedly? What it says if you open <a href="https://reddit.com/r/golang/top.json" rel="nofollow">https://reddit.com/r/golang/top.json</a> in a browser?</p></pre>afhtech: <pre><p>Change
<code>resp, err := http.NewRequest("GET", url, nil)</code></p>
<p>To
<code>resp, err := client.Get(url)</code></p>
<p>Example here
<a href="https://golang.org/pkg/net/http/" rel="nofollow">https://golang.org/pkg/net/http/</a></p></pre>ineedmorealts: <pre><blockquote>
<p>Are you invoking the program repeatedly?</p>
</blockquote>
<p>Nope. I test it ~ 3 time an hour but but the 2nd run I'm getting 429</p>
<blockquote>
<p>What it says if you open <a href="https://reddit.com/r/golang/top.json" rel="nofollow">https://reddit.com/r/golang/top.json</a> in a browser?</p>
</blockquote>
<p>I see the json for the top posts.</p>
<p>This is really starting to bug me because I have written almost the exact same program in python and it works perfectly </p></pre>mikekreuzer: <pre><p>Have you tried the format for the user agent string suggested <a href="https://github.com/reddit/reddit/wiki/API" rel="nofollow">here</a> ?</p></pre>ineedmorealts: <pre><p>I tried and it ran into the same issue</p></pre>ratatask: <pre><p>Maybe you did it wrong ? Update your code so we can see exactly what you're doing.</p></pre>ineedmorealts: <pre><p>It's been updated</p></pre>vendakka: <pre><p>Have you considered creating a developer account and using OAuth? The same top posts query works fine for me when I do that.</p>
<p>I've exported my code as a library so you could try using that and see if it works. You'll need a developer account key.</p>
<p><a href="https://github.com/sridharv/reddit-go" rel="nofollow">github.com/sridharv/reddit-go</a></p></pre>0x6B: <pre><p>You just need to be authenticated (OAUTH) and set a proper 'User-Agent' header.</p>
<p>If you PM me I can sent you a reddit api project (to ugly to be open sourced) of mine which might help you ;-)</p>
<p>edit: Add sample code</p>
<pre><code>package main
import (
"fmt"
"log"
"net/http"
"os"
"strings"
"time"
"golang.org/x/net/context"
"golang.org/x/oauth2"
"golang.org/x/oauth2/clientcredentials"
)
func main() {
cfg := &clientcredentials.Config{
ClientID: "...",
ClientSecret: "...",
TokenURL: "https://www.reddit.com/api/v1/access_token",
Scopes: []string{"read"},
}
ctx := context.Background()
ua := &UserAgent{
Platform: "linux",
AppID: "com.your-domain.your-client-name",
Version: "v0.0.0,
Username: "anon",
}
// This is the only way to set the User-Agent header for the
// way how the oauth2 package aquires the access_token.
// Aaaand reddit is VERY PICKY about the useragent. Even for a single request.
http.DefaultClient.Transport = ua.HTTPTransporter(http.DefaultTransport)
client := &http.Client{
Transport: &oauth2.Transport{
Source: cfg.TokenSource(ctx),
Base: ua.HTTPTransporter(http.DefaultTransport),
},
}
// ...
client.Do(...)
//...
}
// UserAgent contains the information/formating needed for reddit's expectations
// on the 'User-Agent' http handler.
type UserAgent struct {
// Platform of your client i.e "linux"
Platform string
// AppID for your program i.e "com.mydomain.fun-client"
AppID string
// Semantic Version string i.e. "v.0.1.0"
Version string
// Reddit user name i.e "/u/anon"
Username string
}
func (ua *UserAgent) String() string {
return fmt.Sprintf("%s:%s:%s (by %s)",
ua.Platform, ua.AppID, ua.Version, ua.Username)
}
// HTTPTransporter returns a UserAgentTransport which implements the
// http.RoundTripper interface and sets the 'User-Agent' http header.
func (ua *UserAgent) HTTPTransporter(rt http.RoundTripper) *UserAgentTransport {
return &UserAgentTransport{
UserAgent: ua.String(),
RoundTripper: rt,
}
}
type UserAgentTransport struct {
UserAgent string
RoundTripper http.RoundTripper
}
func (t *UserAgentTransport) RoundTrip(req *http.Request) (*http.Response, error) {
req.Header.Set("User-Agent", t.UserAgent)
return t.RoundTripper.RoundTrip(req)
}
</code></pre></pre>fungussa: <pre><p>Your code hasn't logged into reddit.</p>
<pre><code> type RedditResponse struct {
Json struct {
Errors [][]string
Data struct {
Modhash string
Url string
Id string
Name string
}
}
}
....
loginURL := fmt.Sprintf("%sapi/login/%s", s.rootURL, username)
data := url.Values{
"user": {username},
"passwd": {password},
"api_type": {"json"},
}
request, err := http.NewRequest("POST", loginURL, bytes.NewBufferString(data.Encode()))
if err != nil {
return fmt.Errorf("Failed to create a request. err %s\n", err)
}
request.Header.Set("User-Agent", s.defaultUserAgent)
response, err := http.DefaultClient.Do(request)
if err != nil {
return fmt.Errorf("Failed to get the response. %s\n", err)
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return fmt.Errorf("Failed to get the login response: %v\n", response.Status)
}
redditResponse := &RedditResponse{}
err = json.NewDecoder(response.Body).Decode(redditResponse)
if err != nil {
respbytes, _ := ioutil.ReadAll(response.Body)
return fmt.Errorf("Failed to decode the json. err %s. '%v'\n", err, string(respbytes))
}
</code></pre></pre>ineedmorealts: <pre><p>Should it need to log in? I've written pretty much the same program in python and it works fine without having to login </p></pre>fungussa: <pre><p>My app not only reads from, but also writes to Reddit, and I was encountering the same 429 error code. It was only after I'd set the User-Agent header, during the login request, that the 429 error was resolved.</p>
<p>With your app, since it only reads from Reddit, I wouldn't thought that logging in wouldn't be necessary</p></pre>ineedmorealts: <pre><blockquote>
<p>I wouldn't thought that logging in wouldn't be necessary</p>
</blockquote>
<p>Luckily logging is was not required, the issue was fixed when I changed <code>https://reddit.com</code> to <code>httos://www.reddit.com</code></p></pre>gohacker: <pre><p>I've looked into it. Reddit requires you to <code>Set</code> the <code>Host</code> header, too. No need to log in, as the users <a href="/u/fungussa" rel="nofollow">/u/fungussa</a> and <a href="/u/0x6B" rel="nofollow">/u/0x6B</a> claim.</p>
<pre><code>req.Header.Set("User-Agent", "linux:go-postgrabber:v0.1 (by /u/ineedmorealts)")
req.Header.Set("Host", "reddit.com")
</code></pre></pre>fungussa: <pre><p>I didn't need to set the host. Although I had encountered the 429 error code when I hadn't set the User-Agent header</p></pre>ineedmorealts: <pre><p>I set the host header and the issue remains </p></pre>gohacker: <pre><p>Sorry, it appeared to fix the issue for me, but now it doesn't. Even setting the same set of headers wget sets doesn't help. But wget never gets 429. Most strange. Perhaps it's somehow related to the <a href="https://github.com/golang/go/issues/4800" rel="nofollow">issue 4800</a>. Try to change <code>reddit.com</code> to <code>www.reddit.com</code>:</p>
<pre><code>url := fmt.Sprintf("https://www.reddit.com/r/%s/top.json", os.Args[1])`
</code></pre></pre>ineedmorealts: <pre><p>That seems to have fixed it! I still want to do a but more testing to make sure but it seems to be fixed. Thanks!</p></pre>
