Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-17629

Move persisted firewall transaction queue from config to component database for mitigating performance issues



      Both firewall_transaction and firewall_transaction_sequence ( related to Nexus Firewall feature ) are located in the config database. This data is mutated frequently and really belongs in the component database next to the components and assets which are being analysed. Having it in the config database means that database is modified much more often than it needs to be (which has implications for backup and restore) and it's also sync'd separately to the component db changes.

      Having the firewall transaction queue persisted in the config database (rather than the more natural location of the component database) leads to timing issues and inconsistencies under load. This is because changes to the config DB are replicated across the cluster by different OrientDB worker threads compared to the component DB.

      For example, a request to evaluate a new asset (referenced by its encoded entity id) could be queued up on the firewall queue and replicated across the cluster via the config DB workers before that asset record is replicated across the cluster by the component DB workers. Part of this is because we set the quorum to "majority" which means that a change is accepted if a majority of nodes agree with it - this then implies that the minority node could be temporarily inconsistent, but will eventually become consistent. It's during this window that a firewall request could be queued up before all the nodes have the asset record.

      This improvement will move the persisted firewall transaction data to the component database, by changing the current entity adapter to use the component database and by adding an upgrade step to move over existing data. Note that it might not be necessary to move over all data.


          Issue Links



              bradbeck Brad Beck
              mcculls Stuart McCulloch
              Last Updated By:
              Joe Tom Joe Tom
              NXRM - Morpheus
              0 Vote for this issue
              5 Start watching this issue


                Date of First Response: