[asterisk-dev] [Code Review] 3668: refcounter.py uses excessive RAM when processing large refs file

[Corey Farrell]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3668/
-----------------------------------------------------------

(Updated June 25, 2014, 8:22 p.m.)


Review request for Asterisk Developers and Matt Jordan.


Changes
-------

Address Walters findings.

Verified with md5sum that the new code produces the same result as the old code (using a smaller refs log).


Bugs: ASTERISK-23921
    https://issues.asterisk.org/jira/browse/ASTERISK-23921


Repository: Asterisk


Description
-------

When processing a 212MB refs file, refcounter.py used over 3GB of RAM.  This caused swap thrashing and temporarily froze my system.  The included patch makes the following memory optimizations:
* skewed and finished object lists are only populated if not disabled
* lines are saved to each object as the final output line

Saving the whole lines in output format seems to reduce memory usage by 80-90%.  Ignoring finished/skewed objects caused an additional reduction of about 75% on my system.


Diffs (updated)
-----

  /branches/12/contrib/scripts/refcounter.py 417246 

Diff: https://reviewboard.asterisk.org/r/3668/diff/


Testing
-------

Watched 'top -c' with refcounter.py running on the 212MB refs log.  The highest memory usage I saw was 127MB with '-sn' options and 472MB with full output.


Thanks,

Corey Farrell

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20140626/bf336e90/attachment.html>