4 Replies Latest reply: Nov 25, 2015 2:39 PM by Wil Bowes RSS

SteelScript Reporting Functions

Wil Bowes

I've recently started working with SteelScript and Python to help automate some reporting functions. I've had a lot of success with the TrafficSummaryReport and TrafficOverallTimeSeriesReport functions, but am now trying to use the WANTimeSeriesReport code.

 

I've got this far through reading the documentation and looking at the report.py library, but I'm still unable to get any data back from the Profiler:

 

  columns = [p.columns.key.time,

                   p.columns.value.in_avg_bytes,

                   p.columns.value.out_avg_bytes]

 

  timefilter = TimeFilter.parse_range('last 5 min')

 

  report = WANTimeSeriesReport(p)

  report.run(lan_interfaces = [1.2.3.4:6'], wan_interfaces = ['1.2.3.4:7'], direction='inbound', columns=columns, timefilter=timefilter,

                 resolution='auto')

 

  data = report.get_data()

  legend = report.get_legend()

 

  header = [c.key for c in columns]

  Formatter.print_table(data, header)

 

Basically I want to be able to extract average bytes for a SteelHead LAN and WAN interface for each direction to then plot over time.

 

Very keen to hear from anyone who has managed to get this working via SteelScript.

 

Thanks,

 

Wil

  • Re: SteelScript Reporting Functions
    Wil Bowes

    So after some further playing around with this, and getting the WANSummaryReport working, I appreciate that my columns are all wrong for a start... I suspect something close to the following should work, but I'm currently getting a TypeError from pandas:

     

    columns = [p.columns.key.time,
        p.columns.value.avg_bytes,
        p.columns.value.total_bytes]
    
    timefilter = TimeFilter.parse_range('last 1 hour')
    
    # initialize a new report, and run it
    report = WANTimeSeriesReport(p)
    report.run(lan_interfaces=['1.2.3.4:6'], wan_interfaces=['1.2.3.4:7'], direction='outbound', columns=columns, timefilter=timefilter)
    
    # grab the data, and legend (it should be what we passed in for most cases)
    data = report.get_data()
    
    report.delete()
    
    header = [c.key for c in columns]
    Formatter.print_table(data, header)
    

     

    Pandas Error:

     

    Traceback (most recent call last):

      File "ClientGroupReportv1.py", line 102, in <module>

        ExampleApp().run()

      File "C:\...\virtualenv\steelscript\lib\site-packages\steelscript\common\app.py", line 31, in run

        self.parse(sys.argv[1:])

      File "C:\...\virtualenv\steelscript\lib\site-packages\steelscript\commands\steel.py", line 304, in parse

        self.main()

      File "ClientGroupReportv1.py", line 87, in main

        report.run(lan_interfaces=['1.2.3.4:6'], wan_interfaces=['1.2.3.4:7'], direction='outbound', columns=columns, timefilter=timefilter)

      File "C:\...\virtualenv\steelscript\lib\site-packages\steelscript\netprofiler\core\report.py", line 1116, in run

        df_lan.index = df_lan.index.astype(int).astype('M8[s]')

      File "C:\...\virtualenv\steelscript\lib\site-packages\pandas\core\index.py", line 3722, in astype

        self.__class__)

    TypeError: Setting <class 'pandas.core.index.Float64Index'> dtype to anything other than float64 or object is not supported

     

    Understand that this is all probably Python 101 stuff, but any pointers as to where I'm going wrong would be appreciated immensely!

     

    ...and my working WANSummaryReport code is as follows, for anyone who might be interested:

     

    columns = [p.columns.key.idx,
        p.columns.value.avg_bytes,
        p.columns.value.total_bytes]
    
    timefilter = TimeFilter.parse_range('last 1 hour')
    
    # initialize a new report, and run it
    report = WANSummaryReport(p)
    report.run(lan_interfaces = ['1.2.3.4:6'], wan_interfaces = ['1.2.3.4:7'], direction='outbound', columns = columns, timefilter=timefilter)
    
    # grab the data, and legend (it should be what we passed in for most cases)
    data = report.get_data()
    
    report.delete()
    
    header = [c.key for c in columns]
    Formatter.print_table(data, header)
    

     

    Thanks,

     

    Wil

    • Re: Re: SteelScript Reporting Functions
      Wil Bowes

      After further 'jigging' around with my code, I've now discovered that commenting out the following lines in report.py gets me the output I was expecting:

       

      Line 1116: #df_lan.index = df_lan.index.astype(int).astype('M8[s]')
      Line 1120: #df_wan.index = df_wan.index.astype(int).astype('M8[s]')
      

       

      time            avg_bytes        total_bytes
      ------------------------------------------------
      1448406000.0    669508.016667    40170481
      1448406060.0    654733.75        39284025
      1448406120.0    855069.216667    51304153
      1448406180.0    1169004.93333    70140296
      1448406240.0    818760.933333    49125656
      1448406300.0    934541.916667    56072515
      1448406360.0    1155184.71667    69311083
      1448406420.0    801838.066667    48110284
      1448406480.0    795207.933333    47712476
      1448406540.0    792396.333333    47543780
      1448406600.0    878584.316667    52715059
      1448406660.0    974904.283333    58494257
      1448406720.0    811509.483333    48690569
      1448406780.0    656806.35        39408381
      1448406840.0    851054.05        51063243
      1448406900.0    843859.816667    50631589
      

       

      Not sure if it's just my lack of Python skill, or if I've stumbled on a problem with the WANTimeSeriesReport class?

       

      Thanks,

       

      Wil

      • Re: Re: Re: SteelScript Reporting Functions
        Mike Garabedian

        Hi Wil,

         

        Nice analysis!  This is indeed a bug you've stumbled upon, in fact some of the interface discovery methods appear to have issues as well that we will get fixed.

         

        In your last update, commenting out those two lines will leave the time field as just float values, which may be sufficient depending on where you are sending the data.

         

        An alternative method to do a conversion from float to an actual date time object would be the following:

         

        Line 1116: df_lan.index = df_lan.index.map(int).astype('M8[s]')  
        Line 1120: df_wan.index = df_wan.index.map(int).astype('M8[s]') 
        

         

         

        This replaces the first "astype" with a "map" operation.

         

        We will get these issues fixed in our next update to steelscript.netprofiler.

         

        One thing to note about this operation is that its pretty resource intensive since it needs to run up to four reports, this can work nicely when you are targeting one or two interfaces, but might be too much if generating statistics for a fleet of SteelHeads.

         

        Thanks!

        Mike

        • Re: SteelScript Reporting Functions
          Wil Bowes

          Hi Mike,

           

          Thanks for getting back to me. I've been converting epoch to dd-mm-yy hh:mm:ss as a part of my code anyway, so getting it as a float hasn't really impacted me. Will leave the lines commented out and wait for the next release to fix it!

           

          As for resource utilisation on/of the NetProfiler, I assume as I am running my reports sequentially the only constraint is the time they take (and I have available to run them)? Is it possible to impact the operation (i.e. the ability to receive flow or perform analytics) of a NetProfiler through running reports via SteelScript, or are controls in place to mitigate this?

           

          Thanks again,

           

          Wil