我有一个包含许多“视频”信息的JSON . JSON中的每个“视频”都是指向包含“消息”的新JSON的另一个链接 .
我试图遍历“消息”JSON链接并将它们插入MongoDB数据库 .
问题是我得到了一个JSONDecodeError . 我做错了什么,我该如何做对?
回溯(最近一次调用最后一次):文件“/import_messages_dev.py”,第35行,在raw_messages_data = requests.get(url3).json()文件“venv1 / lib / python3.6 / site-packages / requests / models .py“,第892行,在json中返回complexjson.loads(self.text,** kwargs)文件”/usr/lib/python3.6/json/init.py“,第354行,在load中返回_default_decoder.decode( s)文件“/usr/lib/python3.6/json/decoder.py”,第342行,在解码中引发JSONDecodeError(“额外数据”,s,结束)json.decoder.JSONDecodeError:额外数据:第2行第1列(char 380)
import urllib.parse
import requests
import pymongo
### DATABASE ####
# Connect to database // login user:password
uri = 'mongodb://testuser:password@ds245687.mlab.com:45687/liveme'
# Set client.
client = pymongo.MongoClient(uri)
# Set database.
db = client.get_database()
# Create collection.
messages = db['messages']
# The url to the live.me replays.
replay_url = "http://live.ksmobile.net/live/getreplayvideos?"
userid = 895324164037541888
# Parsing the urls for replays and profile with the userid.
url2 = replay_url + urllib.parse.urlencode({'userid': userid}) + '&page_size=1000'
# Printing urls for own validation.
print(f"Replay url: {url2}\n")
# Pull the data from replay json.
raw_replay_data = requests.get(url2).json()
print("Message links: ")
# Insert messages to database.
for i in raw_replay_data['data']['video_info']:
url3 = i['msgfile']
raw_messages_data = requests.get(url3).json()
messages.insert_many(raw_messages_data)
print(url3)
client.close()
Update for further help to the answer
因此,迭代并获取所有链接,逐行读取并将其解析为JSON并将其插入数据库我正在尝试这样做,但它会创建一个新错误 .
for i in raw_replay_data['data']['video_info']:
url3 = i['msgfile']
raw_message_data = urllib.request.urlopen(url3)
for line in raw_message_data:
json_data = json.loads(line)
messages.insert_many(json_data)
新错误是:
Traceback (most recent call last):
File "/import_messages_dev.py", line 54, in <module>
raw_message_data = urllib.request.urlopen(url3)
File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
1 回答
url3
可能包含此值:http://s.live.ksmobile.net/cheetahlive/20/7e/15204559238152116852/15204559238152116852.json包含单个词典但整个文件不存储为JSON数组 .
结构如下:
因此需要逐行读取并将每行解析为JSON .