High CPU OpenEdx on some tasks

Keywords: OpenEdX - Linux - Technical issue - Other
Description:
I’m running bitnami OpenEdx within ProxMox. Server has 8 cores and 64GB of memory. When some tasks within edx are run (e.g., Instructuror–>Student Admin–> View Gradbook the httpd.bin process spikes to 94% constantly…when 40-50 users connect simultaneously and do a quiz, for example, same happens and server does not respond for few mins.
Is there anything I can do to mitigate this as from hardware side, the server should be able to fulfill all the requests without any problem.

thanks.

P.S what I did is that I added the below in: vi /opt/bitnami/apache2/conf/httpd.conf

StartServers 10 MinSpareServers 10 MaxSpareServers 30 = 2.3> MaxRequestWorkers 300 ServerLimit 300 MaxConnectionsPerChild 5000

Is this the right way to increase the concurrent sessions? Anyone has any idea?

Hello @mslyon,

We have a Support Tool that will gather relevant information for us to analyze your configuration and logs. Could you please execute it on the machine where the stack is running by following the steps described in the guide below?

Please note that you need to paste the code ID that is shown at the end.

here it is: 064b3e6f-9f3a-982d-2042-59f29ef367bc
Also, I attche an image of top command…this is when I run a simple quiz on the server…it hits 48% of the CPU…

Hello @mslyon,

There seems to be a few recurring errors from the application itself in the logs:

cat /opt/bitnami/apps/edx/var/log/celery/edx.cms.core.default.log
...
Traceback (most recent call last):
  File "/opt/bitnami/apps/edx/edx-platform/cms/djangoapps/contentstore/tasks.py", line 512, in import_olx
    courselike_items = import_func(
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_importer.py", line 731, in import_course_from_xml
    return list(manager.run_imports())
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_importer.py", line 522, in run_imports
    self.import_children(source_courselike, courselike, courselike_key, dest_id)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_importer.py", line 631, in import_children
    self.recursive_build(source_courselike, courselike, courselike_key, dest_id)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_importer.py", line 479, in recursive_build
    depth_first(source_courselike)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_importer.py", line 477, in depth_first
    depth_first(child)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_importer.py", line 477, in depth_first
    depth_first(child)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_importer.py", line 477, in depth_first
    depth_first(child)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_importer.py", line 464, in depth_first
    _update_and_import_module(
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_importer.py", line 820, in _update_and_import_module
    block = store.import_xblock(
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/mixed.py", line 95, in inner
    retval = func(field_decorator=strip_key_collection, *args, **kwargs)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/mixed.py", line 115, in wrapper
    return func(*args, **kwargs)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/mixed.py", line 788, in import_xblock
    return store.import_xblock(user_id, course_key, block_type, block_id, fields, runtime, **kwargs)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split_draft.py", line 579, in import_xblock
    draft_block = self.import_xblock(user_id, draft_course, block_type, block_id, fields,
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split_draft.py", line 584, in import_xblock
    partitioned_fields = self.partition_fields_by_scope(block_type, fields)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/__init__.py", line 1302, in partition_fields_by_scope
    classes = XBlock.load_class(category, select=prefer_xmodules)
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/xblock/plugin.py", line 109, in load_class
    selected_entry_point = select(identifier, all_entry_points)
  File "/opt/bitnami/apps/edx/edx-platform/common/lib/xmodule/xmodule/modulestore/__init__.py", line 1432, in prefer_xmodules
    return default_select(identifier, entry_points)
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/xblock/plugin.py", line 37, in default_select
    raise PluginMissingError(identifier)
xblock.plugin.PluginMissingError: pdf
[2021-05-15 13:51:22,415: INFO/ForkPoolWorker-1] cms.djangoapps.contentstore.tasks.import_olx[1917afb3-9b14-4080-aa73-2ad6fb4e2361]: Course import course-v1:backup2+test01+UND: Temp data cleared
[2021-05-15 13:51:22,421: ERROR/ForkPoolWorker-1] Unable to queue send_task_complete_email
Traceback (most recent call last):
  File "/opt/bitnami/apps/edx/edx-platform/cms/djangoapps/cms_user_tasks/signals.py", line 52, in user_task_stopped_handler
    send_task_complete_email.delay(status.name.lower(), str(status.state_text), status.user.email, detail_url)
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/celery/app/task.py", line 426, in delay
    return self.apply_async(args, kwargs)
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/celery/app/task.py", line 566, in apply_async
    return app.send_task(
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/celery/app/base.py", line 708, in send_task
    options = router.route(
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/celery/app/routes.py", line 85, in route
    route = self.lookup_route(name, args, kwargs, options, task_type)
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/celery/app/routes.py", line 117, in lookup_route
    route = query(router, name, args, kwargs, options, task_type)
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/celery/app/routes.py", line 122, in query_router
    router = maybe_evaluate(router)
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/kombu/utils/functional.py", line 246, in maybe_evaluate
    return value.evaluate()
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/celery/utils/functional.py", line 51, in evaluate
    self._value = super(mlazy, self).evaluate()
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/kombu/utils/functional.py", line 213, in evaluate
    return self._fun(*self._args, **self._kwargs)
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/celery/app/routes.py", line 130, in expand_router_string
    router = symbol_by_name(router)
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/kombu/utils/imports.py", line 62, in symbol_by_name
    return getattr(module, cls_name) if cls_name else module
  File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/celery/local.py", line 517, in __getattr__
    return ModuleType.__getattribute__(self, name)
AttributeError: module 'celery' has no attribute 'Router'

They don’t seem related to any misconfiguration related to the virtual machine and more to OpenEDX. We recommend you to ask for help in the official forums where someone with more experience can be of help: https://discuss.openedx.org/

Apart from that, it seems our support tool didn’t upload Apache’s logs, probably because of the file size. Could you please share the output of the following command:

sudo tail -60 /opt/bitnami/apache2/logs/error_log

Regards,
Francisco de Paz

thanks but above errors do not have anything to do with the high CPU use.
I experience the same behavior on 2 different servers…I installed a new openEdx (Koaa) on a standalond server and the high CPU is still experienced…

sending:

[Tue May 18 12:00:29.003313 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003318 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003334 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 94, in __call__
[Tue May 18 12:00:29.003339 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = response or self.get_response(request)
[Tue May 18 12:00:29.003354 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003360 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003375 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 94, in __call__
[Tue May 18 12:00:29.003381 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = response or self.get_response(request)
[Tue May 18 12:00:29.003396 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003401 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003417 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 94, in __call__
[Tue May 18 12:00:29.003422 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = response or self.get_response(request)
[Tue May 18 12:00:29.003438 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003449 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003465 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 94, in __call__
[Tue May 18 12:00:29.003471 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = response or self.get_response(request)
[Tue May 18 12:00:29.003486 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003492 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003507 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 94, in __call__
[Tue May 18 12:00:29.003512 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = response or self.get_response(request)
[Tue May 18 12:00:29.003528 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003533 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003548 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 94, in __call__
[Tue May 18 12:00:29.003554 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = response or self.get_response(request)
[Tue May 18 12:00:29.003569 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003574 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003590 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 94, in __call__
[Tue May 18 12:00:29.003595 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = response or self.get_response(request)
[Tue May 18 12:00:29.003611 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003616 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003631 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 94, in __call__
[Tue May 18 12:00:29.003637 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = response or self.get_response(request)
[Tue May 18 12:00:29.003652 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003658 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003673 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 94, in __call__
[Tue May 18 12:00:29.003678 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = response or self.get_response(request)
[Tue May 18 12:00:29.003694 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
[Tue May 18 12:00:29.003703 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = get_response(request)
[Tue May 18 12:00:29.003719 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 93, in __call__
[Tue May 18 12:00:29.003724 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     response = self.process_request(request)
[Tue May 18 12:00:29.003740 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/edx-platform/common/djangoapps/track/middleware.py", line 84, in process_request
[Tue May 18 12:00:29.003745 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     views.server_track(request, request.META['PATH_INFO'], event)
[Tue May 18 12:00:29.003761 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/edx-platform/common/djangoapps/track/views/__init__.py", line 124, in server_track
[Tue May 18 12:00:29.003767 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     eventtracker.emit(name=event_type, data=event)
[Tue May 18 12:00:29.003782 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/eventtracking/tracker.py", line 144, in emit
[Tue May 18 12:00:29.003788 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     return get_tracker().emit(name=name, data=data)
[Tue May 18 12:00:29.003803 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/eventtracking/tracker.py", line 82, in emit
[Tue May 18 12:00:29.003809 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     self.routing_backend.send(event)
[Tue May 18 12:00:29.003824 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/eventtracking/backends/routing.py", line 92, in send
[Tue May 18 12:00:29.003829 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     self.send_to_backends(processed_event)
[Tue May 18 12:00:29.003845 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/eventtracking/backends/routing.py", line 134, in send_to_backends
[Tue May 18 12:00:29.003850 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     backend.send(event)
[Tue May 18 12:00:29.003866 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/eventtracking/backends/routing.py", line 92, in send
[Tue May 18 12:00:29.003871 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     self.send_to_backends(processed_event)
[Tue May 18 12:00:29.003887 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/eventtracking/backends/routing.py", line 134, in send_to_backends
[Tue May 18 12:00:29.003892 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     backend.send(event)
[Tue May 18 12:00:29.003907 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]   File "/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/eventtracking/backends/logger.py", line 42, in send
[Tue May 18 12:00:29.003913 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241]     self.log(event_str)
[Tue May 18 12:00:29.003948 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241] Message: '{"name": "/courses/course-v1:backup2+UND111+UND22/courseware/1fa171efdcbd460cb2ce514d84aea3ef/28fd6f2909444a4b8df0a07722da2ecb/", "context": {"course_id": "course-v1:backup2+UND111+UND22", "course_user_tags": {}, "user_id": 3, "path": "/courses/course-v1:backup2+UND111+UND22/courseware/1fa171efdcbd460cb2ce514d84aea3ef/28fd6f2909444a4b8df0a07722da2ecb/", "org_id": "backup2"}, "username": "XXXX", "session": "7414bb28e926438787bb4a0a1ace61b7", "ip": "93.138.84.XXX", "agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "host": "backup2.xxxx.com", "referer": "https://backup2.xxxx.com/courses/course-v1:backup2+UND111+UND22/course/", "accept_language": "en-US,en;q=0.9,hr;q=0.8,bs;q=0.7,de;q=0.6", "event": "{\\\\"GET\\\\": {\\\\"activate_block_id\\\\": [\\\\"block-v1:backup2+UND111+UND22+type@sequential+block@28fd6f2909444a4b8df0a07722da2ecb\\\\"]}, \\\\"POST\\\\": {}}", "time": "2021-05-18T12:00:29.001442+00:00", "event_type": "/courses/course-v1:backup2+UND111+UND22/courseware/1fa171efdcbd460cb2ce514d84aea3ef/28fd6f2909444a4b8df0a07722da2ecb/", "event_source": "server", "page": null}'
[Tue May 18 12:00:29.003964 2021] [wsgi:error] [pid 26498] [remote 93.138.84.XXX:1241] Arguments: ()

Hello @mslyon,

A constant retry of a failing operation could be causing the CPU spikes. Nevertheless, could you detail the necessary steps to reproduce the spikes in a fresh instance so we can reproduce the issue on our end?

Regards,
Francisco de Paz

Hi thanks but I don’t think so that failing operation causes this.
I tried on a latest fresh installation (Koa) and here is what I did:

  • I created a library with 100 quiz questions inside / then in a new course simply added random question block with 30 questions - after that 40 students connected at once and the server was dead for 5-10 minutes…

What I did now is I configured this /opt/bitnami/apache2/conf/httpd.conf

StartServers 10 MinSpareServers 10 MaxSpareServers 30 = 2.3> MaxRequestWorkers 300 ServerLimit 300 MaxConnectionsPerChild 5000

But not sure if this is the right approach.

In any case you should be able to reproduce is easily with the above steps.

I can send you the library with questions if that can help to do the test on your side?

Thanks.

some more facts:

  • when I run quizzes that come from the library the CPU is around 40-50%
    image

  • when I run the quiz that is not randomized but created directly in the course the spike is around 20%
    image

what do you see in your tests? Do you see same spikes?

Once again: this is a fresh install on a pretty solid server.

another screen that may help

Hello @mslyon,

Thanks for providing the information. I will forward this to my colleagues so they can reproduce these performance tests and get back to you.

Regards,
Francisco de Paz

Hello, just FYI - I have installed tutor also on a brand new server and used Demo course to check the CPU load when running a quiz and I got very much the same high CPU spike:
image

Not sure if this is normal? But in any case when 60 of users connected to my quiz server stopped for 5mins…

Hi @mslyon, note that Open edX is quite a big application, so it is not uncommon to observe CPU spikes. 40% does not seem that big if there are many users making requests at the same time, so increasing the Apache configuration parameters like you did in #10 should be enough.

As for the last message, I can see a mention of uwsgi. Note that don’t use uwsgi for the Bitnami Open edX stack. Is it a custom Python web application?

Hi marcos - nope. I’m the only user. I have a rather nice server (128MB RAM and 6 CPU processor). I was just running a simple quiz (one through demo course and another one from the library as randomized quiz).
uwsgi screenshoot is regarding TUTOR (OpenEdx version) I tried also and I have seen the same behaviour. To me this sounds like some problem with quizzes and edx performance.

If you found the issues with another Open edX distribution, it’s most likely an actual performance issue with Open edX.

The last thing you could do to check, would be to launch an Open edX instance using the official setup scripts, like for instance with the Native Open edX platform installation on Ubuntu 16.04: https://openedx.atlassian.net/wiki/spaces/OpenOPS/pages/146440579/Native%2BOpen%2BedX%2Bplatform%2BUbuntu%2B16.04%2B64%2Bbit%2BInstallation

It will take 1-2 hours to setup and requires a powerful system, but once you have it up and running, you should be able to check if the issue also happens using the official scripts.

Once you confirm that it does, and that this issue is not specific to Bitnami and Tutor, I would recommend to contact official Open edX support channels, like for instance their forum: https://discuss.openedx.org/c/devops/7 - They may be able to give some helpful recommendations to resolve your issues.

I found the issue with Bitnami and with another edx distribution.

Are you also seeing the same issue? Have you tried to do the quick test?

Hi @mslyon, we tried to reproduce the issue by viewing the gradebook but couldn’t find any issues. I guess because we do not have a pre-populated Open edX instance so the tasks are not that complex.

Have you been able to check with a native Open edX platform installation like the ones we mentioned above? If you are able to reproduce the issue, our recommendation is that you contact the developers in the forum (or another location) so they can investigate this.

Probably the best place to report this bug would be in the Community Reported Issues page in their JIRA: https://openedx.atlassian.net/jira/software/c/projects/CRI/issues/

Hi, I did try with native open edx and issue is not present. so all good.

You tested with gradebook - but as explained above, the issue is with the quiz - just try to create a simple quiz with cca 10 questions and you should see it immediately.

thanks.

Hi @mslyon, unfortunately I’m unable to reproduce the issue you mentioned. I just deployed an n1standard-2 instance of Bitnami Open edX on Google Cloud and created a quiz with 10 questions (in the unit, create a new Problem -> Checkboxes 10 times).

When I access the quiz, the page loads fine and top does not show anything out of the ordinary:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 3428 rabbitmq  20   0 2200208  91932   9228 S   3.0   1.2   0:48.76 beam.smp
  414 root      20   0   29652  17092  10524 S   0.3   0.2   0:00.28 unattended-upgr
 3084 mongodb   20   0 1053580  68596  20184 S   0.3   0.9   0:04.86 mongod.bin
 5297 elastic+  20   0 3501808 277780  18700 S   0.3   3.6   0:12.05 java
 5506 daemon    20   0  759692 277240  39836 S   0.3   3.6   0:15.79 .python3.8.bin
 5531 daemon    20   0  759672 277144  39728 S   0.3   3.6   0:15.79 .python3.8.bin
 5557 daemon    20   0  748432 265676  39632 S   0.3   3.5   0:14.89 .python3.8.bin
 5573 daemon    20   0  748184 265556  39716 S   0.3   3.5   0:14.87 .python3.8.bin
 5585 daemon    20   0  131168  38772   7308 S   0.3   0.5   0:00.28 .python3.8.bin
 5894 daemon    20   0 1191924 245592  10204 S   0.3   3.2   0:02.65 .python3.8.bin
 7178 bitnami   20   0   11116   3600   3072 R   0.3   0.0   0:00.30 top

Once I submit a couple of quizes the Python processes will consume some CPU for a couple of seconds but afterwards they will go down to close-to-zero CPU usage.

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 5897 daemon    20   0 1214648 273276  14984 R   5.3   3.6   0:04.18 .python3.8.bin

As you can see it only goes up to 5.3% for me.

Let me know if I’m missing anything. If you were referring to another type of quiz, please provide the details on how to create them in edX Studio.