[asterisk-dev] [Code Review] 3668: refcounter.py uses excessive RAM when processing large refs file

[Corey Farrell]

> On June 25, 2014, 9:54 a.m., wdoekes wrote:
> > /branches/1.8/contrib/scripts/refcounter.py, line 78
> > <https://reviewboard.asterisk.org/r/3668/diff/1/?file=60481#file60481line78>
> >
> >     You could attempt to parse this: a number might hash cheaper than a string.

I'd rather not.  I'm really not that good with Python, the goal of this review is to prevent refcounter.py from using 10x more RAM than the size of the file being processed.  I attempted to avoid adding CPU overhead, but reducing CPU usage is not a goal here.

OTOH if you tell me how to parse this / use a number for the hash key I'm willing to incorporate your suggestion.


> On June 25, 2014, 9:54 a.m., wdoekes wrote:
> > /branches/1.8/contrib/scripts/refcounter.py, lines 66-68
> > <https://reviewboard.asterisk.org/r/3668/diff/1/?file=60481#file60481line66>
> >
> >     You're losing history when addresses are reused.
> >     
> >     That probably does save you memory, but I don't think it's intentional.

Good catch.  Losing history would "save" memory, just like deleting all your family photo's will "save" hard-drive space.

I just did some testing with a smaller refs file (8 MB) and found that revision 1 of this review did drop finished_objects when they reused an address.


- Corey


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3668/#review12311
-----------------------------------------------------------


On June 25, 2014, 8:22 p.m., Corey Farrell wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/3668/
> -----------------------------------------------------------
> 
> (Updated June 25, 2014, 8:22 p.m.)
> 
> 
> Review request for Asterisk Developers and Matt Jordan.
> 
> 
> Bugs: ASTERISK-23921
>     https://issues.asterisk.org/jira/browse/ASTERISK-23921
> 
> 
> Repository: Asterisk
> 
> 
> Description
> -------
> 
> When processing a 212MB refs file, refcounter.py used over 3GB of RAM.  This caused swap thrashing and temporarily froze my system.  The included patch makes the following memory optimizations:
> * skewed and finished object lists are only populated if not disabled
> * lines are saved to each object as the final output line
> 
> Saving the whole lines in output format seems to reduce memory usage by 80-90%.  Ignoring finished/skewed objects caused an additional reduction of about 75% on my system.
> 
> 
> Diffs
> -----
> 
>   /branches/12/contrib/scripts/refcounter.py 417246 
> 
> Diff: https://reviewboard.asterisk.org/r/3668/diff/
> 
> 
> Testing
> -------
> 
> Watched 'top -c' with refcounter.py running on the 212MB refs log.  The highest memory usage I saw was 127MB with '-sn' options and 472MB with full output.
> 
> 
> Thanks,
> 
> Corey Farrell
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20140626/33af4042/attachment-0001.html>