<html>
 <body>
  <div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
   <table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 solid;">
    <tr>
     <td>
      This is an automatically generated e-mail. To reply, visit:
      <a href="https://reviewboard.asterisk.org/r/3668/">https://reviewboard.asterisk.org/r/3668/</a>
     </td>
    </tr>
   </table>
   <br />



 <p>Ship it!</p>



 <pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Some micro-optimization below:</pre>
 <br />







<div>




<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;">
 <thead>
  <tr>
   <th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;">
    <a href="https://reviewboard.asterisk.org/r/3668/diff/3/?file=60999#file60999line84" style="color: black; font-weight: bold; text-decoration: underline;">/branches/12/contrib/scripts/refcounter.py</a>
    <span style="font-weight: normal;">

     (Diff revision 3)

    </span>
   </th>
  </tr>
 </thead>

 <tbody style="background-color: #e4d9cb; padding: 4px 8px; text-align: center;">
  <tr>

   <td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">def process_file(filename):</pre></td>
   <td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">def process_file(options):</pre></td>

  </tr>
 </tbody>



 
 

 <tbody>

  <tr>
    <th bgcolor="#e9eaa8" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">82</font></th>
    <td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">                    <span class="n">skewed_objects</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">current_objects</span><span class="p">[</span><span class="n">obj</span><span class="p">])</span></pre></td>
    <th bgcolor="#e9eaa8" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">84</font></th>
    <td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">                    <span class="n">skewed_objects</span><span class="o">.</span><span class="n">append</span><span class="p">(<span class="hl">[</span></span><span class="n"><span class="hl">obj</span></span><span class="p"><span class="hl">,</span></span><span class="hl"> </span><span class="n">current_objects</span><span class="p">[</span><span class="n">obj</span><span class="p">]<span class="hl">]</span>)</span></pre></td>
  </tr>

 </tbody>

</table>

<pre style="margin-left: 2em; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">> 01:38 < coreyfarrell> anyone who knows python can tell me if I'm
> adding an array to an array the "proper" way?
> mylist.append([ "name1", "value1" ])

Use a tuple for more profit (both speed and memory).

I altered this morning's test to store the items in a list:

$ python thetest.py 
normal: 25.85
edited: 24.80

That last one stores tuples instead of lists. Since they are immutable, they have less overhead. And you won't be mutating them.

Python syntax:

    skewed_objects.append((obj, current_objects[obj]))</pre>
</div>
<br />

<div>




<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;">
 <thead>
  <tr>
   <th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;">
    <a href="https://reviewboard.asterisk.org/r/3668/diff/3/?file=60999#file60999line96" style="color: black; font-weight: bold; text-decoration: underline;">/branches/12/contrib/scripts/refcounter.py</a>
    <span style="font-weight: normal;">

     (Diff revision 3)

    </span>
   </th>
  </tr>
 </thead>

 <tbody style="background-color: #e4d9cb; padding: 4px 8px; text-align: center;">
  <tr>

   <td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">def process_file(filename):</pre></td>
   <td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">def process_file(options):</pre></td>

  </tr>
 </tbody>



 
 

 <tbody>

  <tr>
    <th bgcolor="#e9eaa8" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">87</font></th>
    <td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">                <span class="n">finished_objects</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">lifetime</span><span class="p">)</span></pre></td>
    <th bgcolor="#e9eaa8" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">96</font></th>
    <td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">                    <span class="n">finished_objects</span><span class="o">.</span><span class="n">append</span><span class="p">(<span class="hl">[</span></span><span class="n"><span class="hl">obj</span></span><span class="p"><span class="hl">,</span></span><span class="hl"> </span><span class="n"><span class="hl">current_objects</span></span><span class="p"><span class="hl">[</span></span><span class="n"><span class="hl">obj</span></span><span class="p"><span class="hl">]]</span>)</span></pre></td>
  </tr>

 </tbody>

</table>

<pre style="margin-left: 2em; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">And here.</pre>
</div>
<br />

<div>




<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;">
 <thead>
  <tr>
   <th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;">
    <a href="https://reviewboard.asterisk.org/r/3668/diff/3/?file=60999#file60999line101" style="color: black; font-weight: bold; text-decoration: underline;">/branches/12/contrib/scripts/refcounter.py</a>
    <span style="font-weight: normal;">

     (Diff revision 3)

    </span>
   </th>
  </tr>
 </thead>

 <tbody style="background-color: #e4d9cb; padding: 4px 8px; text-align: center;">
  <tr>

   <td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">def process_file(filename):</pre></td>
   <td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">def process_file(options):</pre></td>

  </tr>
 </tbody>



 
 

 <tbody>

  <tr>
    <th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
    <td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
    <th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">101</font></th>
    <td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">            <span class="n">leaked_objects</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">key</span><span class="p">,</span> <span class="n">lines</span><span class="p">])</span></pre></td>
  </tr>

 </tbody>

</table>

<pre style="margin-left: 2em; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">And here.</pre>
</div>
<br />



<p>- wdoekes</p>


<br />
<p>On June 26th, 2014, 8:06 a.m. UTC, Corey Farrell wrote:</p>








<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" style="background-image: url('https://reviewboard.asterisk.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); background-position: left top; background-repeat: repeat-x; border: 1px black solid;">
 <tr>
  <td>

<div>Review request for Asterisk Developers and Matt Jordan.</div>
<div>By Corey Farrell.</div>


<p style="color: grey;"><i>Updated June 26, 2014, 8:06 a.m.</i></p>







<div style="margin-top: 1.5em;">
 <b style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Bugs: </b>


 <a href="https://issues.asterisk.org/jira/browse/ASTERISK-23921">ASTERISK-23921</a>


</div>



<div style="margin-top: 1.5em;">
 <b style="color: #575012; font-size: 10pt;">Repository: </b>
Asterisk
</div>


<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
 <table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
 <tr>
  <td>
   <pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">When processing a 212MB refs file, refcounter.py used over 3GB of RAM.  This caused swap thrashing and temporarily froze my system.  The included patch makes the following memory optimizations:
* skewed and finished object lists are only populated if not disabled
* lines are saved to each object as the final output line

Saving the whole lines in output format seems to reduce memory usage by 80-90%.  Ignoring finished/skewed objects caused an additional reduction of about 75% on my system.</pre>
  </td>
 </tr>
</table>


<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
 <tr>
  <td>
   <pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Watched 'top -c' with refcounter.py running on the 212MB refs log.  The highest memory usage I saw was 127MB with '-sn' options and 472MB with full output.</pre>
  </td>
 </tr>
</table>


<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">

 <li>/branches/12/contrib/scripts/refcounter.py <span style="color: grey">(417247)</span></li>

</ul>

<p><a href="https://reviewboard.asterisk.org/r/3668/diff/" style="margin-left: 3em;">View Diff</a></p>







  </td>
 </tr>
</table>








  </div>
 </body>
</html>