[asterisk-dev] [Code Review] 3668: refcounter.py uses excessive RAM when processing large refs file

[wdoekes]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3668/#review12341
-----------------------------------------------------------

Ship it!


Some micro-optimization below:


/branches/12/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22533>

    > 01:38 < coreyfarrell> anyone who knows python can tell me if I'm
    > adding an array to an array the "proper" way?
    > mylist.append([ "name1", "value1" ])
    
    Use a tuple for more profit (both speed and memory).
    
    I altered this morning's test to store the items in a list:
    
    $ python thetest.py 
    normal: 25.85
    edited: 24.80
    
    That last one stores tuples instead of lists. Since they are immutable, they have less overhead. And you won't be mutating them.
    
    Python syntax:
    
        skewed_objects.append((obj, current_objects[obj]))



/branches/12/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22534>

    And here.



/branches/12/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22535>

    And here.


- wdoekes


On June 26, 2014, 8:06 a.m., Corey Farrell wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/3668/
> -----------------------------------------------------------
> 
> (Updated June 26, 2014, 8:06 a.m.)
> 
> 
> Review request for Asterisk Developers and Matt Jordan.
> 
> 
> Bugs: ASTERISK-23921
>     https://issues.asterisk.org/jira/browse/ASTERISK-23921
> 
> 
> Repository: Asterisk
> 
> 
> Description
> -------
> 
> When processing a 212MB refs file, refcounter.py used over 3GB of RAM.  This caused swap thrashing and temporarily froze my system.  The included patch makes the following memory optimizations:
> * skewed and finished object lists are only populated if not disabled
> * lines are saved to each object as the final output line
> 
> Saving the whole lines in output format seems to reduce memory usage by 80-90%.  Ignoring finished/skewed objects caused an additional reduction of about 75% on my system.
> 
> 
> Diffs
> -----
> 
>   /branches/12/contrib/scripts/refcounter.py 417247 
> 
> Diff: https://reviewboard.asterisk.org/r/3668/diff/
> 
> 
> Testing
> -------
> 
> Watched 'top -c' with refcounter.py running on the 212MB refs log.  The highest memory usage I saw was 127MB with '-sn' options and 472MB with full output.
> 
> 
> Thanks,
> 
> Corey Farrell
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20140626/9ee5cb8f/attachment-0001.html>