Skip to content

Cant scrape tweets #316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
leon1995 opened this issue Jun 30, 2020 · 5 comments
Closed

Cant scrape tweets #316

leon1995 opened this issue Jun 30, 2020 · 5 comments

Comments

@leon1995
Copy link

When I try to get tweets the following happens:

print(twitterscraper.query_tweets_from_user("elonmusk", 10))
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 77.73.69.56:80
INFO: Retrying... (Attempts left: 50)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 72.175.68.179:80
INFO: Retrying... (Attempts left: 49)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 34.94.110.134:80
INFO: Retrying... (Attempts left: 48)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 115.144.138.178:7804
INFO: Retrying... (Attempts left: 47)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 103.109.56.202:80
INFO: Retrying... (Attempts left: 46)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 20.44.193.208:80
INFO: Retrying... (Attempts left: 45)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 200.255.122.170:8080
INFO: Retrying... (Attempts left: 44)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 203.202.245.62:80
INFO: Retrying... (Attempts left: 43)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 103.28.121.58:3128
INFO: Retrying... (Attempts left: 42)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 34.208.151.217:3838
INFO: Retrying... (Attempts left: 41)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 52.191.103.11:3128
INFO: Retrying... (Attempts left: 40)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 88.198.24.108:8080
INFO: Retrying... (Attempts left: 39)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 161.35.122.145:8080
INFO: Retrying... (Attempts left: 38)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 181.118.167.104:80
INFO: Retrying... (Attempts left: 37)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 132.255.92.34:53281
INFO: Retrying... (Attempts left: 36)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 209.41.69.101:3128
INFO: Retrying... (Attempts left: 35)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 20.43.156.109:80
INFO: Retrying... (Attempts left: 34)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 138.68.240.218:8080
INFO: Retrying... (Attempts left: 33)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 03.9.34.151:3128
INFO: Retrying... (Attempts left: 32)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 88.99.10.248:1080
INFO: Retrying... (Attempts left: 31)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 162.14.18.11:80
INFO: Retrying... (Attempts left: 30)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 176.9.119.170:8080
INFO: Retrying... (Attempts left: 29)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 51.79.173.166:3128
INFO: Retrying... (Attempts left: 28)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 161.202.226.195:8123
INFO: Retrying... (Attempts left: 27)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 161.202.226.194:8123
INFO: Retrying... (Attempts left: 26)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 14.99.23.26:80
INFO: Retrying... (Attempts left: 25)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 187.17.19.90:44985
INFO: Retrying... (Attempts left: 24)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 193.219.169.234:80
INFO: Retrying... (Attempts left: 23)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 51.68.228.131:80
INFO: Retrying... (Attempts left: 22)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 42.3.51.114:80
INFO: Retrying... (Attempts left: 21)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 35.198.31.85:3128
INFO: Retrying... (Attempts left: 20)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 125.167.100.193:80
INFO: Retrying... (Attempts left: 19)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 80.48.119.28:8080
INFO: Retrying... (Attempts left: 18)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 88.99.10.250:1080
INFO: Retrying... (Attempts left: 17)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 104.41.6.224:80
INFO: Retrying... (Attempts left: 16)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 167.172.152.115:3128
INFO: Retrying... (Attempts left: 15)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 34.208.91.142:3838
INFO: Retrying... (Attempts left: 14)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 27.74.254.12:46401
INFO: Retrying... (Attempts left: 13)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 148.72.172.31:8080
INFO: Retrying... (Attempts left: 12)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 47.91.240.187:80
INFO: Retrying... (Attempts left: 11)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 13.85.25.59:8080
INFO: Retrying... (Attempts left: 10)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 187.130.139.197:8080
INFO: Retrying... (Attempts left: 9)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 34.91.135.38:80
INFO: Retrying... (Attempts left: 8)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 187.73.190.117:8080
INFO: Retrying... (Attempts left: 7)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 117.103.205.244:80
INFO: Retrying... (Attempts left: 6)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 51.79.56.11:8080
INFO: Retrying... (Attempts left: 5)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 103.141.64.250:8191
INFO: Retrying... (Attempts left: 4)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 172.104.57.209:666
INFO: Retrying... (Attempts left: 3)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 3.14.120.193:3838
INFO: Retrying... (Attempts left: 2)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 114.7.163.46:8080
INFO: Retrying... (Attempts left: 1)
INFO: Scraping tweets from https://twitter.com/elonmusk
INFO: Using proxy 154.127.36.138:8080
INFO: Got 0 tweets from username elonmusk
[]

Hope that someone can help me :)
@saliudev
Copy link

See this issue:#296

Indeed, this can be fixed by modifying the header dictionary in query.py from
HEADER = {'User-Agent': random.choice(HEADERS_LIST)}
to
HEADER = {'User-Agent': random.choice(HEADERS_LIST), 'X-Requested-With': 'XMLHttpRequest'}
that should fix the issue.

@AmanSavaria1402
Copy link

AmanSavaria1402 commented Jul 1, 2020 •
edited
Loading

This fixed my issue as well, atleast on windows 10.

@sukotyas
Copy link

sukotyas commented Jul 5, 2020

See this issue:#296

Indeed, this can be fixed by modifying the header dictionary in query.py from
HEADER = {'User-Agent': random.choice(HEADERS_LIST)}
to
HEADER = {'User-Agent': random.choice(HEADERS_LIST), 'X-Requested-With': 'XMLHttpRequest'}
that should fix the issue.

this fixed my issue.. thanks..

@AveryData
Copy link

this worked for me too....should it be changed in the git repo here?

@erb13020 erb13020 mentioned this issue Jul 13, 2020
Open
@taspinar
Copy link
Owner

This fix has been merge to twitterscraper and available in version 1.5.0 on pypi as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@taspinar @saliudev @sukotyas @leon1995 @AveryData @AmanSavaria1402