[asterisk-dev] [Code Review] 3668: refcounter.py uses excessive RAM when processing large refs file

[wdoekes]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3668/#review12332
-----------------------------------------------------------

Ship it!


Fire away.


/branches/12/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22508>

    E201 whitespace after '['                                                                                                                                                                     
    E202 whitespace before ']'
    



/branches/12/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22509>

    E201 whitespace after '['                                                                                                                                                                     
    E202 whitespace before ']'
    



/branches/12/contrib/scripts/refcounter.py
<https://reviewboard.asterisk.org/r/3668/#comment22510>

    E201 whitespace after '['                                                                                                                                                                     
    E202 whitespace before ']'
    


P.S. https://github.com/nvie/vim-flake8

sudo pip install flake8
mkdir -p ~/.vim/ftplugin 
wget 'https://raw.githubusercontent.com/nvie/vim-flake8/master/ftplugin/python_flake8.vim' -O ~/.vim/ftplugin/python_flake8.vim
echo 'filetype plugin indent on' >> ~/.vimrc
vim something.py
[press <F7>]



- wdoekes


On June 26, 2014, 12:22 a.m., Corey Farrell wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/3668/
> -----------------------------------------------------------
> 
> (Updated June 26, 2014, 12:22 a.m.)
> 
> 
> Review request for Asterisk Developers and Matt Jordan.
> 
> 
> Bugs: ASTERISK-23921
>     https://issues.asterisk.org/jira/browse/ASTERISK-23921
> 
> 
> Repository: Asterisk
> 
> 
> Description
> -------
> 
> When processing a 212MB refs file, refcounter.py used over 3GB of RAM.  This caused swap thrashing and temporarily froze my system.  The included patch makes the following memory optimizations:
> * skewed and finished object lists are only populated if not disabled
> * lines are saved to each object as the final output line
> 
> Saving the whole lines in output format seems to reduce memory usage by 80-90%.  Ignoring finished/skewed objects caused an additional reduction of about 75% on my system.
> 
> 
> Diffs
> -----
> 
>   /branches/12/contrib/scripts/refcounter.py 417246 
> 
> Diff: https://reviewboard.asterisk.org/r/3668/diff/
> 
> 
> Testing
> -------
> 
> Watched 'top -c' with refcounter.py running on the 212MB refs log.  The highest memory usage I saw was 127MB with '-sn' options and 472MB with full output.
> 
> 
> Thanks,
> 
> Corey Farrell
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20140626/815a7f94/attachment-0001.html>