1Performance Manager 22/12/07 3 4This document will describe an architecture and a phased plan 5for an OpenFabrics OpenIB performance manager. 6 7Currently, there is no open source performance manager, only 8a perfquery diagnostic tool which some have scripted into a 9"poor man's" performance manager. 10 11The primary responsibilities of the performance manager are to: 121. Monitor subnet topology 132. Based on subnet topology, monitor performance and error counters. 14 Also, possible counters related to congestion. 153. Perform data reduction (various calculations (rates, histograms, etc.)) 16 on counters obtained 174. Log performance data and indicate "interesting" related events 18 19 20Performance Manager Components 211. Determine subnet topology 22 Performance manager can determine the subnet topology by subscribing 23 for GID in and out of service events. Upon receipt of a GID in service 24 event, use GID to query SA for corresponding LID by using SubnAdmGet 25 NodeRecord with PortGUID specified. It would utilize the LID and NumPorts 26 returned and add this to the monitoring list. Note that the monitoring 27 list can be extended to be distributed with the manager "balancing" the 28 assignments of new GIDs to the set of known monitors. For GID out of 29 service events, the GID is removed from the monitoring list. 30 312. Monitoring 32 Counters to be monitored include performance counters (data octets and 33 packets both receive and transmit) and error counters. These are all in 34 the mandatory PortCounters attribute. Future support will include the 35 optional 64 bit counters, PortExtendedCounters (as this is only known 36 to be supported on one IB device currently). Also, one congestion 37 counter (PortXmitWait) will also be monitored (on switch ports) initially. 38 39 Polling rather than sampling will be used as the monitoring technique. The 40 polling rate configurable from 1-65535 seconds (default TBD) 41 Note that with 32 bit counters, on 4x SDR links, byte counts can max out in 42 16 seconds and on 4x DDR links in 8 seconds. The polling rate needs to 43 deal with this is accurate byte and packet rates are desired. Since IB 44 counters are sticky, the counters need to be reset when they get "close" 45 to max'ing out. This will result in some inaccuracy. When counters are 46 reset, the time of the reset will be tracked in the monitor and will be 47 queryable. Note that when the 64 bit counters are supported more generally, 48 the polling rate can be reduced. 49 50 The performance manager will support parallel queries. The level of 51 parallelism is configurable with a default of 64 queries outstanding 52 at one time. 53 54 Configuration and dynamic adjustment of any performance manager "knobs" 55 will be supported. 56 57 Also, there will be a console interface to obtain performance data. 58 It will be able to reset counters, report on specific nodes or 59 node types of interest (CAs only, switches only, all, ...). The 60 specifics are TBD. 61 623. Data Reduction 63 For errors, rate rather than raw value will be calculated. Error 64 event is only indicated when rate exceeds a threshold. 65 For packet and byte counters, small changes will be aggregated 66 and only significant changes are updated. 67 Aggregated histograms (per node, all nodes (this is TBD))) for each 68 counter will be provided. Actual counters will also be written to files. 69 NodeGUID will be used to identify node. File formats are TBD. One 70 format to be supported might be CSV. 71 724. Logging 73 "Interesting" events determined by the performance manager will be 74 logged as well as the performance data itself. Significant events 75 will be logged to syslog. There are some interesting scalability 76 issues relative to logging especially for the distributed model. 77 78 Events will be based on rates which are configured as thresholds. 79 There will be configurable thresholds for the error counters with 80 reasonable defaults. Correlation of PerfManager and SM events is 81 interesting but not a mandatory requirement. 82 83 84Performance Manager Scalability 85Clearly as the polling rate goes up, the number of nodes which can be 86monitored from a single performance management node decreases. There is 87some evidence that a single dedicated management node may not be able to 88monitor the largest clusters at a rapid rate. 89 90There are numerous PerfManager models which can be supported: 911. Integrated as thread(s) with OpenSM (run only when SM is master) 922. Standby SM 933. Standalone PerfManager (not running with master or standby SM) 944. Distributed PerfManager (most scalable approach) 95 96Note that these models are in order of implementation complexity and 97hence "schedule". 98 99The simplest model is to run the PerfManager with the master SM. This has 100the least scalability but is the simplest model. Note that in this model 101the topology can be obtained without the GID in and out of service events 102but this is needed for any of the other models to be supported. 103 104The next model is to run the PerfManager with a standby SM. Standbys are not 105doing much currently (polling the master) so there is much idle CPU. 106The downside of this approach is that if the standby takes over as master, 107the PerfManager would need to be moved (or is becomes model 1). 108 109A totally separate standlone PerfManager would allow for a deployment 110model which eliminates the downside of model 2 (standby SM). It could 111still be built in a similar manner with model 2 with unneeded functions 112(SM and SA) not included. The advantage of this model is that it could 113be more readily usable with a vendor specific SM (switch based or otherwise). 114Vendor specific SMs usually come with a built-in performance manager and 115this assumes that there would be a way to disable that performance manager. 116Model 2 can act like model 3 if a disable SM feature is supported in OpenSM 117(command line/console). This will take the SM to not active. 118 119The most scalable model is a distributed PerfManager. One approach to 120distribution is a hierarchial model where there is a PerfManager at the 121top level with a number of PerfMonitors which are responsible for some 122portion of the subnet. 123 124The separation of PerfManager from OpenSM brings up the following additional 125issues: 1261. What communication is needed between OpenSM and the PerfManager ? 1272. Integration of interesting events with OpenSM log 128(Does performance manager assume OpenSM ? Does it need to work with vendor 129SMs ?) 130 131Hierarchial distribution brings up some additional issues: 1321. How is the hierarchy determined ? 1332. How do the PerfManager and PerfMonitors find each other ? 1343. How is the subnet divided amongst the PerfMonitors 1354. Communication amongst the PerfManager and the PerfMonitors 136(including communication failures) 137 138In terms of inter manager communication, there seem to be several 139choices: 1401. Use vendor specific MADs (which can be RMPP'd) and build on top of 141this 1422. Use RC QP communication and build on top of this 1433. Use IPoIB which is much more powerful as sockets can then be utilized 144 145RC QP communication improves on the lower performance of the vendor 146specific MAD approach but is not as powerful as the socket based approach. 147 148The only downside of IPoIB is that it requires multicast to be functioning. 149It seems reasonable to require IPoIB across the management nodes. This 150can either be a separate IPoIB subnet or a shared one with other endnodes 151on the subnet. (If this communication is built on top of sockets, it 152can be any IP subnet amongst the manager nodes). 153 154The first implementation phase will address models 1-3. Model 3 is optional 155as it is similar to models 1 and 2 and may be not be needed. 156 157Model 4 will be addressed in a subsequent implementation phase (and a future 158version of this document). Model 4 can be built on the basis of models 1 and 1592 where some SM, not necessarily master, is the PerfManager and the rest are 160PerfMonitors. 161 162 163Performance Manager Partition Membership 164Note that as the performance manager needs to talk via GSI to the PMAs 165in all the end nodes and GSI utilizes PKey sharing, partition membership 166if invoked must account for this. 167 168The most straightforward deployment of the performance manager is 169to have it be a member of the full default partition (P_Key 0xFFFF). 170 171 172Performance Manager Redundancy 173TBD (future version of this document) 174 175 176Congestion Management 177TBD (future version of this document) 178 179 180QoS Management 181TBD (future version of this document) 182