我在heroku中使用celery的现有python应用程序,我正在尝试迁移到aws beanstalk . 在heroku配置中,我有一个Web worker和芹菜进程 . 我有一个redis插件供芹菜用作经纪人 .

作为迁移的一部分,我还在beanstalk实例上运行redis服务器 . 我正在运行redis和celery,通过beanstalk ebextensions后处理,如某些stackverflow帖子中所推荐的那样 . redis服务器运行正常,我通过redis-cli验证 .

[ec2-user@ip-172-31-3-170 ~]$ ./redis-cli ping
PONG

芹菜也启动并运行工作进程罚款 .

[ec2-user@ip-172-31-3-170 ~]$ ps -ef | grep celery
nobody   20165  2765  0 05:33 ?        00:00:05 /opt/python/run/venv/bin/python run_celery.py -A nekt.tasks worker --loglevel=INFO --maxtasksperchild=2 --autoscale=12,6 --beat
nobody   20171 20165  0 05:33 ?        00:00:00 /opt/python/run/venv/bin/python run_celery.py -A nekt.tasks worker --loglevel=INFO --maxtasksperchild=2 --autoscale=12,6 --beat
nobody   20172 20165  0 05:33 ?        00:00:00 /opt/python/run/venv/bin/python run_celery.py -A nekt.tasks worker --loglevel=INFO --maxtasksperchild=2 --autoscale=12,6 --beat
nobody   20173 20165  0 05:33 ?        00:00:00 /opt/python/run/venv/bin/python run_celery.py -A nekt.tasks worker --loglevel=INFO --maxtasksperchild=2 --autoscale=12,6 --beat
nobody   20174 20165  0 05:33 ?        00:00:00 /opt/python/run/venv/bin/python run_celery.py -A nekt.tasks worker --loglevel=INFO --maxtasksperchild=2 --autoscale=12,6 --beat
nobody   20175 20165  0 05:33 ?        00:00:00 /opt/python/run/venv/bin/python run_celery.py -A nekt.tasks worker --loglevel=INFO --maxtasksperchild=2 --autoscale=12,6 --beat
nobody   20176 20165  0 05:33 ?        00:00:00 /opt/python/run/venv/bin/python run_celery.py -A nekt.tasks worker --loglevel=INFO --maxtasksperchild=2 --autoscale=12,6 --beat
ec2-user 21392 18509  0 06:43 pts/0    00:00:00 grep --color=auto celery

在我的应用程序中,芹菜已经配置为使用REDIS_URL,如下所示(来自我的celery_config.py)

CELERY_TASK_SERIALIZER = "json"
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
BROKER_URL = os.environ['REDIS_URL']
CELERY_RESULT_BACKEND = os.environ['REDIS_URL']

REDIS_URL通过aws beanstalk配置参数定义为redis:// localhost:6379/0

芹菜工作者日志文件表/ var / log显示芹菜连接到本地主机上的redis .

[2016-08-31 05:33:50,772: INFO/MainProcess] Connected to redis://localhost:6379/0
[2016-08-31 05:33:50,778: INFO/MainProcess] mingle: searching for neighbors
[2016-08-31 05:33:51,782: INFO/MainProcess] mingle: all alone
[2016-08-31 05:33:51,788: WARNING/MainProcess] celery@ip-172-31-3-170 ready.

但是,每当从python代码调用celery任务时,它返回套接字错误111 ECONNREFUSED . 这是/ var / log / httpd / error_log的片段

[Wed Aug 31 06:50:48.444874 2016] [:error] [pid 21707] [remote 49.206.126.173:6268] mod_wsgi (pid=21707): Exception occurred processing WSGI script '/opt/python/current/app/manage.py'.
[Wed Aug 31 06:50:48.444919 2016] [:error] [pid 21707] [remote 49.206.126.173:6268] Traceback (most recent call last):
[Wed Aug 31 06:50:48.444939 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
[Wed Aug 31 06:50:48.445330 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     return self.wsgi_app(environ, start_response)
[Wed Aug 31 06:50:48.445343 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
[Wed Aug 31 06:50:48.445359 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     response = self.make_response(self.handle_exception(e))
[Wed Aug 31 06:50:48.445368 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
[Wed Aug 31 06:50:48.445378 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     reraise(exc_type, exc_value, tb)
[Wed Aug 31 06:50:48.445384 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
[Wed Aug 31 06:50:48.445392 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     response = self.full_dispatch_request()
[Wed Aug 31 06:50:48.445397 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
[Wed Aug 31 06:50:48.445406 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     rv = self.handle_user_exception(e)
[Wed Aug 31 06:50:48.445411 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
[Wed Aug 31 06:50:48.445419 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     reraise(exc_type, exc_value, tb)
[Wed Aug 31 06:50:48.445424 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
[Wed Aug 31 06:50:48.445432 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     rv = self.dispatch_request()
[Wed Aug 31 06:50:48.445438 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
[Wed Aug 31 06:50:48.445456 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     return self.view_functions[rule.endpoint](**req.view_args)
[Wed Aug 31 06:50:48.445463 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/current/app/nekt/usercontacts/usercontacts_api.py", line 996, in upload_notify
[Wed Aug 31 06:50:48.445674 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     notify_logs_upload.delay((post_data, result_dict['uid']))
[Wed Aug 31 06:50:48.445686 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/celery/app/task.py", line 453, in delay
[Wed Aug 31 06:50:48.445866 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     return self.apply_async(args, kwargs)
[Wed Aug 31 06:50:48.445876 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/celery/app/task.py", line 565, in apply_async
[Wed Aug 31 06:50:48.445888 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     **dict(self._get_exec_options(), **options)
[Wed Aug 31 06:50:48.445894 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/celery/app/base.py", line 354, in send_task
[Wed Aug 31 06:50:48.446029 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     reply_to=reply_to or self.oid, **options
[Wed Aug 31 06:50:48.446039 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/celery/app/amqp.py", line 310, in publish_task
[Wed Aug 31 06:50:48.446154 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     **kwargs
[Wed Aug 31 06:50:48.446164 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/messaging.py", line 172, in publish
[Wed Aug 31 06:50:48.446288 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     routing_key, mandatory, immediate, exchange, declare)
[Wed Aug 31 06:50:48.446297 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/connection.py", line 457, in _ensured
[Wed Aug 31 06:50:48.446523 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     interval_max)
[Wed Aug 31 06:50:48.446534 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
[Wed Aug 31 06:50:48.446548 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     interval_start, interval_step, interval_max, callback)
[Wed Aug 31 06:50:48.446554 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
[Wed Aug 31 06:50:48.446643 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     return fun(*args, **kwargs)
[Wed Aug 31 06:50:48.446651 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
[Wed Aug 31 06:50:48.446663 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     return self.connection
[Wed Aug 31 06:50:48.446668 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/connection.py", line 742, in connection
[Wed Aug 31 06:50:48.446677 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     self._connection = self._establish_connection()
[Wed Aug 31 06:50:48.446682 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/connection.py", line 697, in _establish_connection
[Wed Aug 31 06:50:48.446690 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     conn = self.transport.establish_connection()
[Wed Aug 31 06:50:48.446695 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
[Wed Aug 31 06:50:48.446755 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     conn = self.Connection(**opts)
[Wed Aug 31 06:50:48.446764 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
[Wed Aug 31 06:50:48.446948 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     self.transport = create_transport(host, connect_timeout, ssl)
[Wed Aug 31 06:50:48.446957 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/amqp/transport.py", line 285, in create_transport
[Wed Aug 31 06:50:48.447024 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     return TCPTransport(host, connect_timeout)
[Wed Aug 31 06:50:48.447034 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]   File "/opt/python/run/venv/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
[Wed Aug 31 06:50:48.447045 2016] [:error] [pid 21707] [remote 49.206.126.173:6268]     raise socket.error(last_err)
[Wed Aug 31 06:50:48.447063 2016] [:error] [pid 21707] [remote 49.206.126.173:6268] error: [Errno 111] ECONNREFUSED

如果我尝试通过从/ opt / python / run / venv / bin运行“celery status”来检查芹菜的状态,那也会返回相同的状态 .

我尝试使用unix域套接字运行redis,使用amazon sqs作为代理,所有这些都给出了相同的错误 . 不知道我错过了什么!