Step 1: Find the Right Sections / Lines of Code to Optimize.
Any pandas calls in "for loops" having over 100 iteration or so, should be addressed first. Further hot pieces can be found using profilers:
- First run something like PyCallGraph (alternative: ProfileEye) which can help you identify the functions taking most amount of time. Further it also shows the break down of where the calls are originating from and outgoing calls, number of calls etc.
- Now apply line profiler on the individual functions identified above to further drill down on specific sections / line(s) that consume most time
Step 2: Optimization Techniques
If you are running only in Linux environment, please check out numba (It uses LLVM to pre-compile code and avoids run time interpretation, giving significant performance boost). Become familiar with pandas functions like expanding_sum / rolling_sum etc. This will not only avoid you writing for-loop (hence elegant code), but also result in better performance. As well, instead of writing for-loop on data frames, use array operation when possible. And if not, python methods like "map" can also give substantial speed (as this will result in interpreter level optimizations of loop).
- Replace as many Pandas call with Numpy call as possible
- Avoid using loc and ix when possible
- Replace iloc with .values[i]
- Use numpy functions for matrix manipulation operation
- Most functions like mean / std / isnan etc. are also supported in numpy with counterparts like mean / nanmean / std / nanstd etc.
- numpy "where" is faster than pandas "where"
- When possible / appropriate, set inplace flag to true in function calls, this will avoid creating copies
Additionally don't rule out Python optimizations (especially in for-loop segments) like avoid dot operator when possible (by assigning the result to a local variable), give preference to local variable over global variables, string formatting instead of string + operator etc.
Some other common mistakes include appending rows in for loop (especially inserting rows up-top), instead append all rows in one go and just expanding the view of the frame.
Converting every single Pandas call to Numpy might not prove very fruitful and would rather end up being and arduous operation and result in some not so good looking code. Instead speeding up critical code segments, and letting the rest be in elegant pandas or your other favorite library will help strike balance between performance and elegance in your code.
Hope these tips help you in your optimization quest!
Rakesh Malpani is a Managing Director at Doran Jones and a senior technologist (Lead developer / Architect) working in the financial markets and technology sectors.