#35408: Optimize post-migrate permission creation
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: contrib.auth | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Adam Johnson:
Old description:
> I have often seen `django.contrib.auth.management.create_permissions()`
> take a significant amount of time in test run profiles. It can be
> optimized by batching more of its operations, including making
> `ContentTypeManager.get_for_models()` use batch creation.
>
> For a comparison, I profiled 1518 of Django’s 1518 tests in modules
> called “models”:
>
> {{{
> $ python -m cProfile -o profile runtests.py --parallel 1 *model*
>
> $ python -m pstats profile <<< 'sort cumtime
> stats 10000' | less
> }}}
>
> Before optimization stats:
>
> * Total 11,938,857 function calls taking 5.349 seconds.
> * 88 calls to `create_permissions()` take 456ms, ~8.5% of the total time.
>
> After optimization stats:
>
> * Total 11,359,071 function calls taking 5.035 seconds.
> * 88 calls to `create_permissions()` now take 239ms, ~4.7% of the toal
> time.
> * 217ms and 579,786 function calls saved.
>
> Optimization is limited because the `post_migrate` signal runs once per
> migrated app config, so there’s no chance to bulk create *all* content
> types and permissions at once. If we introduced a new “all migrated apps”
> signal, that could reduce runtime further by batching all creation.
New description:
I have often seen `django.contrib.auth.management.create_permissions()`
take a significant amount of time in test run profiles. It can be
optimized by batching more of its operations, including making
`ContentTypeManager.get_for_models()` use batch creation.
For a comparison, I profiled 1518 of Django’s tests in modules called
“models”:
{{{
$ python -m cProfile -o profile runtests.py --parallel 1 *model*
$ python -m pstats profile <<< 'sort cumtime
stats 10000' | less
}}}
Before optimization stats:
* Total 11,938,857 function calls taking 5.349 seconds.
* 88 calls to `create_permissions()` take 456ms, ~8.5% of the total time.
After optimization stats:
* Total 11,359,071 function calls taking 5.035 seconds.
* 88 calls to `create_permissions()` now take 239ms, ~4.7% of the toal
time.
* 217ms and 579,786 function calls saved.
Optimization is limited because the `post_migrate` signal runs once per
migrated app config, so there’s no chance to bulk create *all* content
types and permissions at once. If we introduced a new “all migrated apps”
signal, that could reduce runtime further by batching all creation.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/35408#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/0107018f622fbded-e615ad31-74a2-4e3a-8cda-e59d13132830-000000%40eu-central-1.amazonses.com.
