[asterisk-dev] [Code Review] 3668: refcounter.py uses excessive RAM when processing large refs file

[wdoekes]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3668/#review12311
-----------------------------------------------------------


Making the storage optional: good.
Storing strings: good, apparently it saves you a lot of mem.

But turning storage lists into dicts is probably not the way to go.


/branches/1.8/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22489>

    You're losing history when addresses are reused.
    
    That probably does save you memory, but I don't think it's intentional.



/branches/1.8/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22490>

    You could attempt to parse this: a number might hash cheaper than a string.



/branches/1.8/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22487>

    PEP nazi says: E126 continuation line over-indented for hanging indent
    
    Go with either:
    
                current_objects[obj].append("[%s] %s:%s %s: %s %s - [%s]" %
                                            (parsed_line['thread_id'],
                                             parsed_line['file'],
                                             parsed_line['line'],
    
    Or:
    
                current_objects[obj].append("[%s] %s:%s %s: %s %s - [%s]" % (
                    parsed_line['thread_id'],
                    parsed_line['file'],
                    parsed_line['line'],
    



/branches/1.8/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22488>

    No need for the temporary lifetime variable.
    
    And current_objects.get(obj) can be replaced with current_objects[obj] since it exists. (Or we'd get a KeyError upon deleting it.)
    


- wdoekes


On June 24, 2014, 5:17 a.m., Corey Farrell wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/3668/
> -----------------------------------------------------------
> 
> (Updated June 24, 2014, 5:17 a.m.)
> 
> 
> Review request for Asterisk Developers and Matt Jordan.
> 
> 
> Bugs: ASTERISK-23921
>     https://issues.asterisk.org/jira/browse/ASTERISK-23921
> 
> 
> Repository: Asterisk
> 
> 
> Description
> -------
> 
> When processing a 212MB refs file, refcounter.py used over 3GB of RAM.  This caused swap thrashing and temporarily froze my system.  The included patch makes the following memory optimizations:
> * skewed and finished object lists are only populated if not disabled
> * lines are saved to each object as the final output line
> 
> Saving the whole lines in output format seems to reduce memory usage by 80-90%.  Ignoring finished/skewed objects caused an additional reduction of about 75% on my system.
> 
> 
> Diffs
> -----
> 
>   /branches/1.8/contrib/scripts/refcounter.py 417058 
> 
> Diff: https://reviewboard.asterisk.org/r/3668/diff/
> 
> 
> Testing
> -------
> 
> Watched 'top -c' with refcounter.py running on the 212MB refs log.  The highest memory usage I saw was 127MB with '-sn' options and 472MB with full output.
> 
> 
> Thanks,
> 
> Corey Farrell
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20140625/ea15d7c9/attachment-0001.html>