3 Replies Latest reply: Mar 31, 2016 5:27 AM by jchessman RSS

10K UI Limit, removed with script?

Jamie Lozan

Hi, I ran a few reports using the "trafficsummary.py" script that is provided with steelscript.  I was under the impression that programmatically pulling this data allowed you to go beyond the 10,000 line limit that is imposed in the GUI of NetProfiler?  I've pulled various data sets and each time it ends at line 10003 (first few lines are labels and empty).


Here's the syntax I was using:

"python traffic_summary.py *** -u *** -p *** --groupby host_pair_protoport --timefilter "10/22/14 7:00 to 10/28/14 18:00" --trafficexpr "(src hostgroup ByLocationSummary:az and dst hostgroup ByLocation:NDC) or (src hostgroup ByLocationSummary:az and dst hostgroup ByLocation:MDC)" --columns cli_group_name,cli_host_ip,srv_group_name,srv_host_ip,protoport,total_bytes >> c:\users\lozja001\documents\export4.csv"


Is there a way to set the data resolution and the "any, within, outside" criteria inside this traffic expression or another option?


Thank you!

  • Re: 10K UI Limit, removed with script?



    I'm not sure entirely what you are asking, about the 10k row limit or the data resolution/any/within/outside criteria so I will answer both.


    For the 10k row limit I am not sure this has been built in to SteelScript yet. I know it is possible directly through the REST API (I will attach a LINUX BASH script to this message that does it) but I have not seen anything in SteelScript yet.  The script takes four CLI parameters: "username> <password> <NetProfiler IP> <number of rows to return> (in that order) and returns all the matching data.  The data is unformatted so the script is not very useful other then as an example.


    For your second set of questions, you can definitely change the resolution using the "resolution" parameter. The traffic_summary.pl script only supports using 'auto' so you can not change it with the default script (at least not the one I have - it may have changed at some point).  I've modified that script (and attached it below) to support a '--resolution' option. This can be used to specify the desired resolution: auto, 15min, 1hour, 6hour, 1day, 1week.


    Finally, about using any, within, outside.  These specific options are not supported because you are assumed to be writing a traffic expression. In reality all the UI does is takes the data you put in the various boxes and converts it to a traffic expression.  So if you put in the port/proto box Outside and tcp/443 that is just writing the traffic expression "(not port 443 and not proto tcp)".  To do WIthin and tcp/443 the expression would be "(port 443 and proto tcp)".  Finally, any would be blank (so if you don't specify a traffic expression the expectation is that everything is included.  A more complex example would be:


    Host A: Within, Server

    Host B: Outside, Client

    Port/Proto: Within TCP/443, TCP/80


    The traffic expression for this would be:

    "srv host and not cli host and (port 443 and proto tcp) and (port 80 and proto tcp)"


    I hope this helps!



  • Re: 10K UI Limit, removed with script?

    Hi Jamie-


    I wanted to post an update to this topic.The most recent version of SteelScript has added support for the limit function. If you look at the docs/tutorial (specifically 4.4. NetProfiler and Reporting — steelscript documentation) you will see that there is now a 'limit' option for get_data:


    get_data(index=0, columns=None, limit=None)

    Retrieve data for this report.

    If columns is specified, restrict the data to the list of requested columns.

    Parameters:limit (integer) – Upper limit of rows of the result data.

    If you specify limit it will retrieve that number of rows and if you leave it blank you will get 10,000.