[asterisk-dev] [Code Review] 3668: refcounter.py uses excessive RAM when processing large refs file

[Corey Farrell]

> On June 26, 2014, 11:57 a.m., wdoekes wrote:
> > Some micro-optimization below:

Thanks for teaching me this about Python!


- Corey


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3668/#review12341
-----------------------------------------------------------


On June 27, 2014, 3:14 p.m., Corey Farrell wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/3668/
> -----------------------------------------------------------
> 
> (Updated June 27, 2014, 3:14 p.m.)
> 
> 
> Review request for Asterisk Developers and Matt Jordan.
> 
> 
> Bugs: ASTERISK-23921
>     https://issues.asterisk.org/jira/browse/ASTERISK-23921
> 
> 
> Repository: Asterisk
> 
> 
> Description
> -------
> 
> When processing a 212MB refs file, refcounter.py used over 3GB of RAM.  This caused swap thrashing and temporarily froze my system.  The included patch makes the following memory optimizations:
> * skewed and finished object lists are only populated if not disabled
> * lines are saved to each object as the final output line
> 
> Saving the whole lines in output format seems to reduce memory usage by 80-90%.  Ignoring finished/skewed objects caused an additional reduction of about 75% on my system.
> 
> 
> Diffs
> -----
> 
>   /branches/12/contrib/scripts/refcounter.py 417247 
> 
> Diff: https://reviewboard.asterisk.org/r/3668/diff/
> 
> 
> Testing
> -------
> 
> Watched 'top -c' with refcounter.py running on the 212MB refs log.  The highest memory usage I saw was 127MB with '-sn' options and 472MB with full output.
> 
> 
> Thanks,
> 
> Corey Farrell
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20140627/1c7755de/attachment.html>