From 864b11e4c54d6d02bb0dd27c5017c8ace7f58845 Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Mon, 20 Nov 2017 16:10:50 +0100 Subject: [PATCH 01/10] new readme --- README.rst | 593 +---------------------------------------------------- 1 file changed, 1 insertion(+), 592 deletions(-) diff --git a/README.rst b/README.rst index 529b361..4fc168b 100644 --- a/README.rst +++ b/README.rst @@ -1,595 +1,4 @@ django-netjsongraph =================== -.. image:: https://travis-ci.org/netjson/django-netjsongraph.svg - :target: https://travis-ci.org/netjson/django-netjsongraph - -.. image:: https://coveralls.io/repos/netjson/django-netjsongraph/badge.svg - :target: https://coveralls.io/r/netjson/django-netjsongraph - -.. image:: https://requires.io/github/netjson/django-netjsongraph/requirements.svg?branch=master - :target: https://requires.io/github/netjson/django-netjsongraph/requirements/?branch=master - :alt: Requirements Status - -.. image:: https://badge.fury.io/py/django-netjsongraph.svg - :target: http://badge.fury.io/py/django-netjsongraph - ------------- - -Reusable django app for collecting and visualizing network topology. - -.. image:: https://raw.githubusercontent.com/netjson/django-netjsongraph/master/docs/images/visualizer.png - -.. image:: https://raw.githubusercontent.com/netjson/django-netjsongraph/master/docs/images/admin.png - -.. contents:: **Table of Contents**: - :backlinks: none - :depth: 3 - ------------- - -Current features ----------------- - -* **network topology collector** supporting different formats: - - NetJSON NetworkGraph - - OLSR (jsoninfo/txtinfo) - - batman-adv (jsondoc/txtinfo) - - BMX6 (q6m) - - CNML 1.0 - - additional formats can be added by `specifying custom parsers <#netjsongraph-parsers>`_ -* **network topology visualizer** based on `netjsongraph.js `_ -* **simple HTTP API** that exposes data in `NetJSON `__ *NetworkGraph* format -* **admin interface** that allows to easily manage, audit, visualize and debug topologies and their relative data (nodes, links) -* **receive topology** from multiple nodes - -Project goals -------------- - -* make it easy to visualize network topology data for the formats supported by `netdiff `_ -* expose topology data via RESTful resources in *NetJSON NetworkGraph* format -* make it easy to integrate in larger django projects to improve reusability -* make it easy to extend its models by providing abstract models (**needs improvement in this point**) -* provide ways to customize or replace the visualizer (**needs improvement in this point**) -* keep the core very simple -* provide ways to extend the default behaviour -* encourage new features to be published as extensions - -Deploy it in production ------------------------ - -An automated installer is provided by the `OpenWISP `_ project: -`ansible-openwisp2 `_. - -Ensure to follow the instructions explained in the following section: `Enabling the network topology -module `_. - -Install stable version from pypi --------------------------------- - -Install from pypi: - -.. code-block:: shell - - pip install django-netjsongraph - -Install development version ---------------------------- - -Install tarball: - -.. code-block:: shell - - pip install https://github.com/netjson/django-netjsongraph/tarball/master - -Alternatively you can install via pip using git: - -.. code-block:: shell - - pip install -e git+git://github.com/netjson/django-netjsongraph#egg=django-netjsongraph - -If you want to contribute, install your cloned fork: - -.. code-block:: shell - - git clone git@github.com:/django-netjsongraph.git - cd django-netjsongraph - python setup.py develop - -Setup (integrate in an existing django project) ------------------------------------------------ - -Add ``rest_framework`` and ``django_netjsongraph`` to ``INSTALLED_APPS``: - -.. code-block:: python - - INSTALLED_APPS = [ - # other apps - 'rest_framework', - 'openwisp_utils.admin_theme', - 'django_netjsongraph' - # ... - ] - -Include urls in your urlconf (you can change the prefixes -according to your needs): - -.. code-block:: python - - from django.conf.urls import include, url - - from django_netjsongraph.api import urls as netjsongraph_api - from django_netjsongraph.visualizer import urls as netjsongraph_visualizer - - urlpatterns = [ - # your URLs ... - url(r'^api/', include(netjsongraph_api)), - url(r'', include(netjsongraph_visualizer)), - ] - -Create database tables:: - - ./manage.py migrate - -Management Commands -------------------- - -``update_topology`` -^^^^^^^^^^^^^^^^^^^ - -After topology URLs (URLs exposing the files that the topology of the network) have been -added in the admin, the ``update_topology`` management command can be used to collect data -and start playing with the network graph:: - - ./manage.py update_topology - -The management command accepts a ``--label`` argument that will be used to search in -topology labels, eg:: - - ./manage.py update_topology --label mytopology - -Logging -------- - -The ``update_topology`` management command will automatically try to log errors. - -For a good default ``LOGGING`` configuration refer to the `test settings -`_. - -Strategies ----------- - -There are mainly two ways of collecting topology information: - -* **FETCH** strategy -* **RECEIVE** strategy - -Each ``Topology`` instance has a ``strategy`` field which can be set to the desired setting. - -FETCH strategy -^^^^^^^^^^^^^^ - -Topology data will be fetched from a URL. - -When some links are not detected anymore they will be flagged as "down" straightaway. - -RECEIVE strategy -^^^^^^^^^^^^^^^^ - -Topology data is sent directly from one or more nodes of the network. - -The collector waits to receive data in the payload of a POST HTTP request; -when such a request is received, a ``key`` parameter it's first checked against -the ``Topology`` key. - -If the request is authorized the collector proceeds to update the topology. - -If the data is sent from one node only, it's highly advised to set the -``expiration_time`` of the ``Topology`` instance to ``0`` (seconds), this way the -system works just like in the **FETCH strategy**, with the only difference that -the data is sent by one node instead of fetched by the collector. - -If the data is sent from multiple nodes, you **SHOULD** set the ``expiration_time`` -of the ``Topology`` instance to a value slightly higher than the interval used -by nodes to send the topology, this way links will be flagged as "down" only if -they haven't been detected for a while. This mechanism allows to visualize the -topology even if the network has been split in several parts, the disadvantage -is that it will take a bit more time to detect links that go offline. - -Settings --------- - -``NETJSONGRAPH_PARSERS`` -^^^^^^^^^^^^^^^^^^^^^^^^ - -+--------------+-------------+ -| **type**: | ``list`` | -+--------------+-------------+ -| **default**: | ``[]`` | -+--------------+-------------+ - -Additional custom `netdiff parsers `_. - -``NETJSONGRAPH_SIGNALS`` -^^^^^^^^^^^^^^^^^^^^^^^^ - -+--------------+-------------+ -| **type**: | ``str`` | -+--------------+-------------+ -| **default**: | ``None`` | -+--------------+-------------+ - -String representing python module to import on initialization. - -Useful for loading django signals or to define custom behaviour. - -``NETJSONGRAPH_TIMEOUT`` -^^^^^^^^^^^^^^^^^^^^^^^^ - -+--------------+-------------+ -| **type**: | ``int`` | -+--------------+-------------+ -| **default**: | ``8`` | -+--------------+-------------+ - -Timeout when fetching topology URLs. - -``NETJSONGRAPH_LINK_EXPIRATION`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -+--------------+-------------+ -| **type**: | ``int`` | -+--------------+-------------+ -| **default**: | ``60`` | -+--------------+-------------+ - -If a link is down for more days than this number, it will be deleted by the -``update_topology`` management command. - -Setting this to ``False`` will disable this feature. - -``NETJSONGRAPH_VISUALIZER_CSS`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -+--------------+--------------------------------+ -| **type**: | ``str`` | -+--------------+--------------------------------+ -| **default**: | ``netjsongraph/css/style.css`` | -+--------------+--------------------------------+ - -Path of the visualizer css file. Allows customization of css according to user's -preferences. - -Overriding visualizer templates -------------------------------- - -Follow these steps to override and customise the visualizer's default templates: - -* create a directory in your django project and put its full path in ``TEMPLATES['DIRS']``, - which can be found in the django ``settings.py`` file -* create a sub directory named ``netjsongraph`` and add all the templates which shall override - the default ``netjsongraph/*`` templates -* create a template file with the same name of the template file you want to override - -More information about the syntax used in django templates can be found in the `django templates -documentation `_. - -Example: overriding the `` - -Extending django-netjsongraph ------------------------------ - -*django-netjsongraph* provides a set of models, admin classes and generic views which can be imported, extended and reused by third party apps. - -To extend *django-netjsongraph*, **you MUST NOT** add it to ``settings.INSTALLED_APPS``, but you must create your own app (which goes into ``settings.INSTALLED_APPS``), import the base classes from django-netjsongraph and add your customizations. - -Extending models -^^^^^^^^^^^^^^^^ - -This example provides an example of how to extend the base models of -*django-netjsongraph*. - -.. code-block:: python - - # models.py of your custom ``network`` app - from django.db import models - - from django_netjsongraph.base.link import AbstractLink - from django_netjsongraph.base.node import AbstractNode - from django_netjsongraph.base.topology import AbstractTopology - - # the model ``organizations.Organization`` is omitted for brevity - # if you are curious to see a real implementation, check out django-organizations - # https://github.com/bennylope/django-organizations - - class OrganizationMixin(models.Model): - organization = models.ForeignKey('organization.Organization') - - class Meta: - abstract = True - - - class Topology(OrganizationMixin, AbstractTopology): - def clean(self): - # your own validation logic here - pass - - class Meta(AbstractTopology.Meta): - abstract = False - - - class Node(AbstractNode): - topology = models.ForeignKey('Topology') - - class Meta: - abstract = False - - - class Link(AbstractLink): - topology = models.ForeignKey('Topology') - source = models.ForeignKey('Node', - related_name='source_link_set') - target = models.ForeignKey('Node', - related_name='source_target_set') - - class Meta: - abstract = False - -Extending the admin -^^^^^^^^^^^^^^^^^^^ - -Following the above example, you can avoid duplicating the admin code by importing the base admin classes and registering your models with. - -.. code-block:: python - - # admin.py of your app - from django.contrib import admin - from django_netjsongraph.base.admin import (AbstractLinkAdmin, - AbstractNodeAdmin, - AbstractTopologyAdmin) - # these are you custom models - from .models import Link, Node, Topology - - - class TopologyAdmin(AbstractTopologyAdmin): - model = Topology - - - class NodeAdmin(AbstractNodeAdmin): - model = Node - - - class LinkAdmin(AbstractLinkAdmin): - model = Link - - - admin.site.register(Link, LinkAdmin) - admin.site.register(Node, NodeAdmin) - admin.site.register(Topology, TopologyAdmin) - -Extending API views -^^^^^^^^^^^^^^^^^^^ - -If your use case doesn't vary much from the base, you may also want to try to reuse the API views: - -.. code-block:: python - - # your app.api.views - from ..models import Topology - from django_netjsongraph.api.generics import (BaseNetworkCollectionView, - BaseNetworkGraphView, - BaseReceiveTopologyView) - - - class NetworkCollectionView(BaseNetworkCollectionView): - queryset = Topology.objects.filter(published=True) - - - class NetworkGraphView(BaseNetworkGraphView): - queryset = Topology.objects.filter(published=True) - - - class ReceiveTopologyView(BaseReceiveTopologyView): - model = Topology - - - network_collection = NetworkCollectionView.as_view() - network_graph = NetworkGraphView.as_view() - receive_topology = ReceiveTopologyView.as_view() - -API URLs -^^^^^^^^ - -If you are not making drastic changes to the api views, you can avoid duplicating the URL logic by using the ``get_api_urls`` function. Put this in your api ``urls.py``: - -.. code-block:: python - - # your app.api.urls - from django_netjsongraph.utils import get_api_urls - from . import views - - urlpatterns = get_api_urls(views) - -Extending Visualizer Views -^^^^^^^^^^^^^^^^^^^^^^^^^^ -If your use case doesn't vary much from the base, you may also want to try to reuse the Visualizer views: - -.. code-block:: python - - # your app.visualizer.views - from ..models import Topology - from .generics import BaseTopologyDetailView, BaseTopologyListView - - - class TopologyListView(BaseTopologyListView): - topology_model = Topology - - - class TopologyDetailView(BaseTopologyDetailView): - topology_model = Topology - - - topology_list = TopologyListView.as_view() - topology_detail = TopologyDetailView.as_view() - - -Visualizer URLs -^^^^^^^^^^^^^^^ -If you are not making any drastic changes to visualizer views, you can avoid duplicating the URL logic by using ``get_visualizer_urls`` function. Put this in your visualizer ``urls.py`` - -.. code-block:: python - - # your app.visualizer.urls - from django_netjsongraph.utils import get_visualizer_urls - from . import views - - urlpatterns = get_visualizer_urls(views) - -Extending AppConfig -^^^^^^^^^^^^^^^^^^^ - -You may want to reuse the ``AppConfig`` class of *django-netjsongraph* too: - -.. code-block:: python - - from django_netjsongraph.apps import DjangoNetjsongraphConfig - - class MyOwnConfig(DjangoNetjsongraphConfig): - name = 'yourapp' - label = 'yourapp' - -Installing for development --------------------------- - -Install sqlite: - -.. code-block:: shell - - sudo apt-get install sqlite3 libsqlite3-dev - -Install your forked repo: - -.. code-block:: shell - - git clone git://github.com//django-netjsongraph - cd django-netjsongraph/ - python setup.py develop - -Install test requirements: - -.. code-block:: shell - - pip install -r requirements-test.txt - -Create database: - -.. code-block:: shell - - cd tests/ - ./manage.py migrate - ./manage.py createsuperuser - -Launch development server: - -.. code-block:: shell - - ./manage.py runserver - -You can access the visualizer at http://127.0.0.1:8000/ -and the admin interface at http://127.0.0.1:8000/admin/. - -Run tests with: - -.. code-block:: shell - - ./runtests.py - -Contributing ------------- - -First off, thanks for taking the time to read these guidelines. - -Trying to follow these guidelines is important in order to minimize waste and -avoid misunderstandings. - -1. Ensure your changes meet the `Project Goals`_ -2. If you found a bug please send a failing test with a patch -3. If you want to add a new feature, announce your intentions in the - `issue tracker `_ -4. Fork this repo and install it by following the instructions in - `Installing for development`_ -5. Follow `PEP8, Style Guide for Python Code`_ -6. Write code -7. Write tests for your code -8. Ensure all tests pass -9. Ensure test coverage is not under 90% -10. Document your changes -11. Send pull request - -.. _PEP8, Style Guide for Python Code: http://www.python.org/dev/peps/pep-0008/ -.. _ninux-dev mailing list: http://ml.ninux.org/mailman/listinfo/ninux-dev - -Changelog ---------- - -See `CHANGES `_. - -License -------- - -See `LICENSE `_. - -This projects bundles third-party javascript libraries in its source code: - -- `D3.js (BSD-3-Clause) `_ +This is a fork of the original django-netjsongraph project that implements the modifications developed in the `netCommons `_ project. \ No newline at end of file From 249b3a6c98a8b2bc95283e14a0fdeeed76162e3b Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Mon, 20 Nov 2017 16:40:32 +0100 Subject: [PATCH 02/10] added kind field to snapshots --- django_netjsongraph/base/snapshot.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/django_netjsongraph/base/snapshot.py b/django_netjsongraph/base/snapshot.py index 2792ed0..bdec8ca 100644 --- a/django_netjsongraph/base/snapshot.py +++ b/django_netjsongraph/base/snapshot.py @@ -3,6 +3,11 @@ from .base import TimeStampedEditableModel +KINDS = ( + ('normal', _('NORMAL')), + ('block_cut_tree', _('BLOCK_CUT_TREE')) +) + class AbstractSnapshot(TimeStampedEditableModel): """ @@ -11,6 +16,11 @@ class AbstractSnapshot(TimeStampedEditableModel): topology = models.ForeignKey('django_netjsongraph.topology') data = models.TextField(blank=False) date = models.DateField(auto_now=True) + kind = models.CharField(_('kind'), + max_length=16, + choices=KINDS, + default='normal', + db_index=True) class Meta: verbose_name_plural = _('snapshots') From 5b1d3206fc6c114d992751cd83e272b776c97a3e Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Mon, 20 Nov 2017 18:47:33 +0100 Subject: [PATCH 03/10] generate and save more than one snapshot per topology, now raises an error for multiple topologies, to be fixed with correct default in api/generic.py --- django_netjsongraph/base/snapshot.py | 10 ++---- django_netjsongraph/base/topology.py | 48 ++++++++++++++++++++++------ 2 files changed, 41 insertions(+), 17 deletions(-) diff --git a/django_netjsongraph/base/snapshot.py b/django_netjsongraph/base/snapshot.py index bdec8ca..aca79fc 100644 --- a/django_netjsongraph/base/snapshot.py +++ b/django_netjsongraph/base/snapshot.py @@ -2,11 +2,7 @@ from django.utils.translation import ugettext_lazy as _ from .base import TimeStampedEditableModel - -KINDS = ( - ('normal', _('NORMAL')), - ('block_cut_tree', _('BLOCK_CUT_TREE')) -) +from .topology import SNAPSHOT_KINDS class AbstractSnapshot(TimeStampedEditableModel): @@ -18,8 +14,8 @@ class AbstractSnapshot(TimeStampedEditableModel): date = models.DateField(auto_now=True) kind = models.CharField(_('kind'), max_length=16, - choices=KINDS, - default='normal', + choices=SNAPSHOT_KINDS, + default=SNAPSHOT_KINDS[0], db_index=True) class Meta: diff --git a/django_netjsongraph/base/topology.py b/django_netjsongraph/base/topology.py index 802e2e5..e07649c 100644 --- a/django_netjsongraph/base/topology.py +++ b/django_netjsongraph/base/topology.py @@ -12,6 +12,9 @@ from django.utils.translation import ugettext_lazy as _ from netdiff import NetJsonParser, diff from rest_framework.utils.encoders import JSONEncoder +from netdiff.utils import _netjson_networkgraph as to_netjson +from netjson_robustness.analyser import ParsedGraph + from ..contextmanagers import log_failure from ..settings import PARSERS, TIMEOUT @@ -23,6 +26,13 @@ ('receive', _('RECEIVE')) ) +# TODO export these in the configuration +# First element is default +SNAPSHOT_KINDS = ( + ('normal', _('NORMAL')), + ('block_cut_tree', _('BLOCK_CUT_TREE')) +) + @python_2_unicode_compatible class AbstractTopology(TimeStampedEditableModel): @@ -254,16 +264,34 @@ def save_snapshot(self, **kwargs): """ Saves the snapshot of topology """ - Snapshot = self.snapshot_model - date = datetime.now().date() - options = dict(topology=self, date=date) - options.update(kwargs) - try: - s = Snapshot.objects.get(**options) - except: - s = Snapshot(**options) - s.data = self.json() - s.save() + for kind in SNAPSHOT_KINDS: + Snapshot = self.snapshot_model + date = datetime.now().date() + options = dict(topology=self, date=date, kind=kind[0]) + options.update(kwargs) + try: + s = Snapshot.objects.get(**options) + except: + s = Snapshot(**options) + + if kind[0] == "block_cut_tree": + s.data = self.compute_cut_block_tree() + elif kind[0] == "normal": + s.data = self.json() + s.save() + + def compute_cut_block_tree(self): + """ + transform a given topology in a cut-block tree for better + visualization of the critical nodes + """ + netjson = NetJsonParser(self.json(dict=True, omit_down=True)) + p = ParsedGraph(netjson) + p.condensate_graph() + return json.dumps(to_netjson(p.netJSON.protocol, p.netJSON.version, + p.netJSON.revision, p.netJSON.metric, + p.condensed_graph.nodes(data=True), + p.condensed_graph.edges(data=True), dict=True)) def link_status_changed(self, link, status): """ From 6d458cfc965f51d47ce61d1a22e0039f61b2411b Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Tue, 21 Nov 2017 09:25:59 +0100 Subject: [PATCH 04/10] added default value for visualizations --- django_netjsongraph/api/generics.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/django_netjsongraph/api/generics.py b/django_netjsongraph/api/generics.py index 31a33a0..7fae835 100644 --- a/django_netjsongraph/api/generics.py +++ b/django_netjsongraph/api/generics.py @@ -37,7 +37,10 @@ class BaseNetworkGraphHistoryView(APIView): def get(self, request, pk, format=None): topology = get_object_or_404(self.topology_model, pk) date = request.query_params.get('date') - options = dict(topology=topology, date=date) + kind = request.query_params.get('kind') + if not kind: + kind = "normal" + options = dict(topology=topology, date=date, kind=kind) # missing date: 400 if not date: return Response({'detail': _('missing required "date" parameter')}, From 84c9168568d76707de1cce67d1215e6399ece1f7 Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Tue, 21 Nov 2017 19:21:35 +0100 Subject: [PATCH 05/10] introduced radiobutton for the choice of the visualization --- .../static/netjsongraph/js/topology-history.js | 8 +++++++- .../templates/netjsongraph/netjsongraph-content.html | 8 +++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/django_netjsongraph/static/netjsongraph/js/topology-history.js b/django_netjsongraph/static/netjsongraph/js/topology-history.js index 2dcf2d9..a30539b 100644 --- a/django_netjsongraph/static/netjsongraph/js/topology-history.js +++ b/django_netjsongraph/static/netjsongraph/js/topology-history.js @@ -2,16 +2,22 @@ window.initTopologyHistory = function($){ var datepicker = $('#njg-datepicker'), today = new Date(), apiUrl = datepicker.attr('data-history-api'); + var kindpicker=$("input[name='visualization_kind']"); + var kind = 'normal'; + kindpicker.change(function() {; + kind = this.value + }); today.setHours(0, 0, 0, 0); datepicker.datepicker({dateFormat: 'dd/mm/yy'}); datepicker.datepicker('setDate', today); datepicker.change(function() {; var date = datepicker.val().split('/').reverse().join('-'), - url = apiUrl + '?date=' + date; + url = apiUrl + '?kind=' + kind + '&date=' + date; // load latest data when looking currentDate if(datepicker.datepicker('getDate').getTime() == today.getTime()){ url = window.__njg_default_url__; } + // load latest data when looking currentDate $.getJSON(url).done(function(data){ window.graph = window.loadNetJsonGraph(data); }).error(function(xhr){ diff --git a/django_netjsongraph/templates/netjsongraph/netjsongraph-content.html b/django_netjsongraph/templates/netjsongraph/netjsongraph-content.html index d94cd52..cfe4855 100644 --- a/django_netjsongraph/templates/netjsongraph/netjsongraph-content.html +++ b/django_netjsongraph/templates/netjsongraph/netjsongraph-content.html @@ -4,7 +4,13 @@

  {% trans 'link down' %}

+
+ Choose Visualization + data-history-api="{{ history_url }}">
+
+
+
+
From 36b7461ebaa500dc7bb97b36616e74651ad03556 Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Mon, 20 Nov 2017 16:40:32 +0100 Subject: [PATCH 06/10] added kind field to snapshots --- django_netjsongraph/base/snapshot.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/django_netjsongraph/base/snapshot.py b/django_netjsongraph/base/snapshot.py index 2792ed0..bdec8ca 100644 --- a/django_netjsongraph/base/snapshot.py +++ b/django_netjsongraph/base/snapshot.py @@ -3,6 +3,11 @@ from .base import TimeStampedEditableModel +KINDS = ( + ('normal', _('NORMAL')), + ('block_cut_tree', _('BLOCK_CUT_TREE')) +) + class AbstractSnapshot(TimeStampedEditableModel): """ @@ -11,6 +16,11 @@ class AbstractSnapshot(TimeStampedEditableModel): topology = models.ForeignKey('django_netjsongraph.topology') data = models.TextField(blank=False) date = models.DateField(auto_now=True) + kind = models.CharField(_('kind'), + max_length=16, + choices=KINDS, + default='normal', + db_index=True) class Meta: verbose_name_plural = _('snapshots') From 083e303fac1d3332308b10c64d811500e768a67a Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Mon, 20 Nov 2017 18:47:33 +0100 Subject: [PATCH 07/10] generate and save more than one snapshot per topology, now raises an error for multiple topologies, to be fixed with correct default in api/generic.py --- django_netjsongraph/base/snapshot.py | 10 ++---- django_netjsongraph/base/topology.py | 48 ++++++++++++++++++++++------ 2 files changed, 41 insertions(+), 17 deletions(-) diff --git a/django_netjsongraph/base/snapshot.py b/django_netjsongraph/base/snapshot.py index bdec8ca..aca79fc 100644 --- a/django_netjsongraph/base/snapshot.py +++ b/django_netjsongraph/base/snapshot.py @@ -2,11 +2,7 @@ from django.utils.translation import ugettext_lazy as _ from .base import TimeStampedEditableModel - -KINDS = ( - ('normal', _('NORMAL')), - ('block_cut_tree', _('BLOCK_CUT_TREE')) -) +from .topology import SNAPSHOT_KINDS class AbstractSnapshot(TimeStampedEditableModel): @@ -18,8 +14,8 @@ class AbstractSnapshot(TimeStampedEditableModel): date = models.DateField(auto_now=True) kind = models.CharField(_('kind'), max_length=16, - choices=KINDS, - default='normal', + choices=SNAPSHOT_KINDS, + default=SNAPSHOT_KINDS[0], db_index=True) class Meta: diff --git a/django_netjsongraph/base/topology.py b/django_netjsongraph/base/topology.py index 802e2e5..e07649c 100644 --- a/django_netjsongraph/base/topology.py +++ b/django_netjsongraph/base/topology.py @@ -12,6 +12,9 @@ from django.utils.translation import ugettext_lazy as _ from netdiff import NetJsonParser, diff from rest_framework.utils.encoders import JSONEncoder +from netdiff.utils import _netjson_networkgraph as to_netjson +from netjson_robustness.analyser import ParsedGraph + from ..contextmanagers import log_failure from ..settings import PARSERS, TIMEOUT @@ -23,6 +26,13 @@ ('receive', _('RECEIVE')) ) +# TODO export these in the configuration +# First element is default +SNAPSHOT_KINDS = ( + ('normal', _('NORMAL')), + ('block_cut_tree', _('BLOCK_CUT_TREE')) +) + @python_2_unicode_compatible class AbstractTopology(TimeStampedEditableModel): @@ -254,16 +264,34 @@ def save_snapshot(self, **kwargs): """ Saves the snapshot of topology """ - Snapshot = self.snapshot_model - date = datetime.now().date() - options = dict(topology=self, date=date) - options.update(kwargs) - try: - s = Snapshot.objects.get(**options) - except: - s = Snapshot(**options) - s.data = self.json() - s.save() + for kind in SNAPSHOT_KINDS: + Snapshot = self.snapshot_model + date = datetime.now().date() + options = dict(topology=self, date=date, kind=kind[0]) + options.update(kwargs) + try: + s = Snapshot.objects.get(**options) + except: + s = Snapshot(**options) + + if kind[0] == "block_cut_tree": + s.data = self.compute_cut_block_tree() + elif kind[0] == "normal": + s.data = self.json() + s.save() + + def compute_cut_block_tree(self): + """ + transform a given topology in a cut-block tree for better + visualization of the critical nodes + """ + netjson = NetJsonParser(self.json(dict=True, omit_down=True)) + p = ParsedGraph(netjson) + p.condensate_graph() + return json.dumps(to_netjson(p.netJSON.protocol, p.netJSON.version, + p.netJSON.revision, p.netJSON.metric, + p.condensed_graph.nodes(data=True), + p.condensed_graph.edges(data=True), dict=True)) def link_status_changed(self, link, status): """ From ec48c6ea111e6311a7e0b48f6d68c1f6d4fa3b84 Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Tue, 21 Nov 2017 09:25:59 +0100 Subject: [PATCH 08/10] added default value for visualizations --- django_netjsongraph/api/generics.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/django_netjsongraph/api/generics.py b/django_netjsongraph/api/generics.py index 31a33a0..7fae835 100644 --- a/django_netjsongraph/api/generics.py +++ b/django_netjsongraph/api/generics.py @@ -37,7 +37,10 @@ class BaseNetworkGraphHistoryView(APIView): def get(self, request, pk, format=None): topology = get_object_or_404(self.topology_model, pk) date = request.query_params.get('date') - options = dict(topology=topology, date=date) + kind = request.query_params.get('kind') + if not kind: + kind = "normal" + options = dict(topology=topology, date=date, kind=kind) # missing date: 400 if not date: return Response({'detail': _('missing required "date" parameter')}, From 29897f614d22d0aef70ac17f8fdc35900825cd36 Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Tue, 21 Nov 2017 19:21:35 +0100 Subject: [PATCH 09/10] introduced radiobutton for the choice of the visualization --- .../static/netjsongraph/js/topology-history.js | 8 +++++++- .../templates/netjsongraph/netjsongraph-content.html | 8 +++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/django_netjsongraph/static/netjsongraph/js/topology-history.js b/django_netjsongraph/static/netjsongraph/js/topology-history.js index 2dcf2d9..a30539b 100644 --- a/django_netjsongraph/static/netjsongraph/js/topology-history.js +++ b/django_netjsongraph/static/netjsongraph/js/topology-history.js @@ -2,16 +2,22 @@ window.initTopologyHistory = function($){ var datepicker = $('#njg-datepicker'), today = new Date(), apiUrl = datepicker.attr('data-history-api'); + var kindpicker=$("input[name='visualization_kind']"); + var kind = 'normal'; + kindpicker.change(function() {; + kind = this.value + }); today.setHours(0, 0, 0, 0); datepicker.datepicker({dateFormat: 'dd/mm/yy'}); datepicker.datepicker('setDate', today); datepicker.change(function() {; var date = datepicker.val().split('/').reverse().join('-'), - url = apiUrl + '?date=' + date; + url = apiUrl + '?kind=' + kind + '&date=' + date; // load latest data when looking currentDate if(datepicker.datepicker('getDate').getTime() == today.getTime()){ url = window.__njg_default_url__; } + // load latest data when looking currentDate $.getJSON(url).done(function(data){ window.graph = window.loadNetJsonGraph(data); }).error(function(xhr){ diff --git a/django_netjsongraph/templates/netjsongraph/netjsongraph-content.html b/django_netjsongraph/templates/netjsongraph/netjsongraph-content.html index d94cd52..cfe4855 100644 --- a/django_netjsongraph/templates/netjsongraph/netjsongraph-content.html +++ b/django_netjsongraph/templates/netjsongraph/netjsongraph-content.html @@ -4,7 +4,13 @@

  {% trans 'link down' %}

+
+ Choose Visualization + data-history-api="{{ history_url }}">
+
+
+
+
From ad79c9968e1ba64f5eed03b7a2b0696e25dbb9cf Mon Sep 17 00:00:00 2001 From: Leonardo Maccari Date: Fri, 21 Dec 2018 16:09:34 +0100 Subject: [PATCH 10/10] restore original README --- README.rst | 593 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 592 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 4fc168b..529b361 100644 --- a/README.rst +++ b/README.rst @@ -1,4 +1,595 @@ django-netjsongraph =================== -This is a fork of the original django-netjsongraph project that implements the modifications developed in the `netCommons `_ project. \ No newline at end of file +.. image:: https://travis-ci.org/netjson/django-netjsongraph.svg + :target: https://travis-ci.org/netjson/django-netjsongraph + +.. image:: https://coveralls.io/repos/netjson/django-netjsongraph/badge.svg + :target: https://coveralls.io/r/netjson/django-netjsongraph + +.. image:: https://requires.io/github/netjson/django-netjsongraph/requirements.svg?branch=master + :target: https://requires.io/github/netjson/django-netjsongraph/requirements/?branch=master + :alt: Requirements Status + +.. image:: https://badge.fury.io/py/django-netjsongraph.svg + :target: http://badge.fury.io/py/django-netjsongraph + +------------ + +Reusable django app for collecting and visualizing network topology. + +.. image:: https://raw.githubusercontent.com/netjson/django-netjsongraph/master/docs/images/visualizer.png + +.. image:: https://raw.githubusercontent.com/netjson/django-netjsongraph/master/docs/images/admin.png + +.. contents:: **Table of Contents**: + :backlinks: none + :depth: 3 + +------------ + +Current features +---------------- + +* **network topology collector** supporting different formats: + - NetJSON NetworkGraph + - OLSR (jsoninfo/txtinfo) + - batman-adv (jsondoc/txtinfo) + - BMX6 (q6m) + - CNML 1.0 + - additional formats can be added by `specifying custom parsers <#netjsongraph-parsers>`_ +* **network topology visualizer** based on `netjsongraph.js `_ +* **simple HTTP API** that exposes data in `NetJSON `__ *NetworkGraph* format +* **admin interface** that allows to easily manage, audit, visualize and debug topologies and their relative data (nodes, links) +* **receive topology** from multiple nodes + +Project goals +------------- + +* make it easy to visualize network topology data for the formats supported by `netdiff `_ +* expose topology data via RESTful resources in *NetJSON NetworkGraph* format +* make it easy to integrate in larger django projects to improve reusability +* make it easy to extend its models by providing abstract models (**needs improvement in this point**) +* provide ways to customize or replace the visualizer (**needs improvement in this point**) +* keep the core very simple +* provide ways to extend the default behaviour +* encourage new features to be published as extensions + +Deploy it in production +----------------------- + +An automated installer is provided by the `OpenWISP `_ project: +`ansible-openwisp2 `_. + +Ensure to follow the instructions explained in the following section: `Enabling the network topology +module `_. + +Install stable version from pypi +-------------------------------- + +Install from pypi: + +.. code-block:: shell + + pip install django-netjsongraph + +Install development version +--------------------------- + +Install tarball: + +.. code-block:: shell + + pip install https://github.com/netjson/django-netjsongraph/tarball/master + +Alternatively you can install via pip using git: + +.. code-block:: shell + + pip install -e git+git://github.com/netjson/django-netjsongraph#egg=django-netjsongraph + +If you want to contribute, install your cloned fork: + +.. code-block:: shell + + git clone git@github.com:/django-netjsongraph.git + cd django-netjsongraph + python setup.py develop + +Setup (integrate in an existing django project) +----------------------------------------------- + +Add ``rest_framework`` and ``django_netjsongraph`` to ``INSTALLED_APPS``: + +.. code-block:: python + + INSTALLED_APPS = [ + # other apps + 'rest_framework', + 'openwisp_utils.admin_theme', + 'django_netjsongraph' + # ... + ] + +Include urls in your urlconf (you can change the prefixes +according to your needs): + +.. code-block:: python + + from django.conf.urls import include, url + + from django_netjsongraph.api import urls as netjsongraph_api + from django_netjsongraph.visualizer import urls as netjsongraph_visualizer + + urlpatterns = [ + # your URLs ... + url(r'^api/', include(netjsongraph_api)), + url(r'', include(netjsongraph_visualizer)), + ] + +Create database tables:: + + ./manage.py migrate + +Management Commands +------------------- + +``update_topology`` +^^^^^^^^^^^^^^^^^^^ + +After topology URLs (URLs exposing the files that the topology of the network) have been +added in the admin, the ``update_topology`` management command can be used to collect data +and start playing with the network graph:: + + ./manage.py update_topology + +The management command accepts a ``--label`` argument that will be used to search in +topology labels, eg:: + + ./manage.py update_topology --label mytopology + +Logging +------- + +The ``update_topology`` management command will automatically try to log errors. + +For a good default ``LOGGING`` configuration refer to the `test settings +`_. + +Strategies +---------- + +There are mainly two ways of collecting topology information: + +* **FETCH** strategy +* **RECEIVE** strategy + +Each ``Topology`` instance has a ``strategy`` field which can be set to the desired setting. + +FETCH strategy +^^^^^^^^^^^^^^ + +Topology data will be fetched from a URL. + +When some links are not detected anymore they will be flagged as "down" straightaway. + +RECEIVE strategy +^^^^^^^^^^^^^^^^ + +Topology data is sent directly from one or more nodes of the network. + +The collector waits to receive data in the payload of a POST HTTP request; +when such a request is received, a ``key`` parameter it's first checked against +the ``Topology`` key. + +If the request is authorized the collector proceeds to update the topology. + +If the data is sent from one node only, it's highly advised to set the +``expiration_time`` of the ``Topology`` instance to ``0`` (seconds), this way the +system works just like in the **FETCH strategy**, with the only difference that +the data is sent by one node instead of fetched by the collector. + +If the data is sent from multiple nodes, you **SHOULD** set the ``expiration_time`` +of the ``Topology`` instance to a value slightly higher than the interval used +by nodes to send the topology, this way links will be flagged as "down" only if +they haven't been detected for a while. This mechanism allows to visualize the +topology even if the network has been split in several parts, the disadvantage +is that it will take a bit more time to detect links that go offline. + +Settings +-------- + +``NETJSONGRAPH_PARSERS`` +^^^^^^^^^^^^^^^^^^^^^^^^ + ++--------------+-------------+ +| **type**: | ``list`` | ++--------------+-------------+ +| **default**: | ``[]`` | ++--------------+-------------+ + +Additional custom `netdiff parsers `_. + +``NETJSONGRAPH_SIGNALS`` +^^^^^^^^^^^^^^^^^^^^^^^^ + ++--------------+-------------+ +| **type**: | ``str`` | ++--------------+-------------+ +| **default**: | ``None`` | ++--------------+-------------+ + +String representing python module to import on initialization. + +Useful for loading django signals or to define custom behaviour. + +``NETJSONGRAPH_TIMEOUT`` +^^^^^^^^^^^^^^^^^^^^^^^^ + ++--------------+-------------+ +| **type**: | ``int`` | ++--------------+-------------+ +| **default**: | ``8`` | ++--------------+-------------+ + +Timeout when fetching topology URLs. + +``NETJSONGRAPH_LINK_EXPIRATION`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++--------------+-------------+ +| **type**: | ``int`` | ++--------------+-------------+ +| **default**: | ``60`` | ++--------------+-------------+ + +If a link is down for more days than this number, it will be deleted by the +``update_topology`` management command. + +Setting this to ``False`` will disable this feature. + +``NETJSONGRAPH_VISUALIZER_CSS`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++--------------+--------------------------------+ +| **type**: | ``str`` | ++--------------+--------------------------------+ +| **default**: | ``netjsongraph/css/style.css`` | ++--------------+--------------------------------+ + +Path of the visualizer css file. Allows customization of css according to user's +preferences. + +Overriding visualizer templates +------------------------------- + +Follow these steps to override and customise the visualizer's default templates: + +* create a directory in your django project and put its full path in ``TEMPLATES['DIRS']``, + which can be found in the django ``settings.py`` file +* create a sub directory named ``netjsongraph`` and add all the templates which shall override + the default ``netjsongraph/*`` templates +* create a template file with the same name of the template file you want to override + +More information about the syntax used in django templates can be found in the `django templates +documentation `_. + +Example: overriding the `` + +Extending django-netjsongraph +----------------------------- + +*django-netjsongraph* provides a set of models, admin classes and generic views which can be imported, extended and reused by third party apps. + +To extend *django-netjsongraph*, **you MUST NOT** add it to ``settings.INSTALLED_APPS``, but you must create your own app (which goes into ``settings.INSTALLED_APPS``), import the base classes from django-netjsongraph and add your customizations. + +Extending models +^^^^^^^^^^^^^^^^ + +This example provides an example of how to extend the base models of +*django-netjsongraph*. + +.. code-block:: python + + # models.py of your custom ``network`` app + from django.db import models + + from django_netjsongraph.base.link import AbstractLink + from django_netjsongraph.base.node import AbstractNode + from django_netjsongraph.base.topology import AbstractTopology + + # the model ``organizations.Organization`` is omitted for brevity + # if you are curious to see a real implementation, check out django-organizations + # https://github.com/bennylope/django-organizations + + class OrganizationMixin(models.Model): + organization = models.ForeignKey('organization.Organization') + + class Meta: + abstract = True + + + class Topology(OrganizationMixin, AbstractTopology): + def clean(self): + # your own validation logic here + pass + + class Meta(AbstractTopology.Meta): + abstract = False + + + class Node(AbstractNode): + topology = models.ForeignKey('Topology') + + class Meta: + abstract = False + + + class Link(AbstractLink): + topology = models.ForeignKey('Topology') + source = models.ForeignKey('Node', + related_name='source_link_set') + target = models.ForeignKey('Node', + related_name='source_target_set') + + class Meta: + abstract = False + +Extending the admin +^^^^^^^^^^^^^^^^^^^ + +Following the above example, you can avoid duplicating the admin code by importing the base admin classes and registering your models with. + +.. code-block:: python + + # admin.py of your app + from django.contrib import admin + from django_netjsongraph.base.admin import (AbstractLinkAdmin, + AbstractNodeAdmin, + AbstractTopologyAdmin) + # these are you custom models + from .models import Link, Node, Topology + + + class TopologyAdmin(AbstractTopologyAdmin): + model = Topology + + + class NodeAdmin(AbstractNodeAdmin): + model = Node + + + class LinkAdmin(AbstractLinkAdmin): + model = Link + + + admin.site.register(Link, LinkAdmin) + admin.site.register(Node, NodeAdmin) + admin.site.register(Topology, TopologyAdmin) + +Extending API views +^^^^^^^^^^^^^^^^^^^ + +If your use case doesn't vary much from the base, you may also want to try to reuse the API views: + +.. code-block:: python + + # your app.api.views + from ..models import Topology + from django_netjsongraph.api.generics import (BaseNetworkCollectionView, + BaseNetworkGraphView, + BaseReceiveTopologyView) + + + class NetworkCollectionView(BaseNetworkCollectionView): + queryset = Topology.objects.filter(published=True) + + + class NetworkGraphView(BaseNetworkGraphView): + queryset = Topology.objects.filter(published=True) + + + class ReceiveTopologyView(BaseReceiveTopologyView): + model = Topology + + + network_collection = NetworkCollectionView.as_view() + network_graph = NetworkGraphView.as_view() + receive_topology = ReceiveTopologyView.as_view() + +API URLs +^^^^^^^^ + +If you are not making drastic changes to the api views, you can avoid duplicating the URL logic by using the ``get_api_urls`` function. Put this in your api ``urls.py``: + +.. code-block:: python + + # your app.api.urls + from django_netjsongraph.utils import get_api_urls + from . import views + + urlpatterns = get_api_urls(views) + +Extending Visualizer Views +^^^^^^^^^^^^^^^^^^^^^^^^^^ +If your use case doesn't vary much from the base, you may also want to try to reuse the Visualizer views: + +.. code-block:: python + + # your app.visualizer.views + from ..models import Topology + from .generics import BaseTopologyDetailView, BaseTopologyListView + + + class TopologyListView(BaseTopologyListView): + topology_model = Topology + + + class TopologyDetailView(BaseTopologyDetailView): + topology_model = Topology + + + topology_list = TopologyListView.as_view() + topology_detail = TopologyDetailView.as_view() + + +Visualizer URLs +^^^^^^^^^^^^^^^ +If you are not making any drastic changes to visualizer views, you can avoid duplicating the URL logic by using ``get_visualizer_urls`` function. Put this in your visualizer ``urls.py`` + +.. code-block:: python + + # your app.visualizer.urls + from django_netjsongraph.utils import get_visualizer_urls + from . import views + + urlpatterns = get_visualizer_urls(views) + +Extending AppConfig +^^^^^^^^^^^^^^^^^^^ + +You may want to reuse the ``AppConfig`` class of *django-netjsongraph* too: + +.. code-block:: python + + from django_netjsongraph.apps import DjangoNetjsongraphConfig + + class MyOwnConfig(DjangoNetjsongraphConfig): + name = 'yourapp' + label = 'yourapp' + +Installing for development +-------------------------- + +Install sqlite: + +.. code-block:: shell + + sudo apt-get install sqlite3 libsqlite3-dev + +Install your forked repo: + +.. code-block:: shell + + git clone git://github.com//django-netjsongraph + cd django-netjsongraph/ + python setup.py develop + +Install test requirements: + +.. code-block:: shell + + pip install -r requirements-test.txt + +Create database: + +.. code-block:: shell + + cd tests/ + ./manage.py migrate + ./manage.py createsuperuser + +Launch development server: + +.. code-block:: shell + + ./manage.py runserver + +You can access the visualizer at http://127.0.0.1:8000/ +and the admin interface at http://127.0.0.1:8000/admin/. + +Run tests with: + +.. code-block:: shell + + ./runtests.py + +Contributing +------------ + +First off, thanks for taking the time to read these guidelines. + +Trying to follow these guidelines is important in order to minimize waste and +avoid misunderstandings. + +1. Ensure your changes meet the `Project Goals`_ +2. If you found a bug please send a failing test with a patch +3. If you want to add a new feature, announce your intentions in the + `issue tracker `_ +4. Fork this repo and install it by following the instructions in + `Installing for development`_ +5. Follow `PEP8, Style Guide for Python Code`_ +6. Write code +7. Write tests for your code +8. Ensure all tests pass +9. Ensure test coverage is not under 90% +10. Document your changes +11. Send pull request + +.. _PEP8, Style Guide for Python Code: http://www.python.org/dev/peps/pep-0008/ +.. _ninux-dev mailing list: http://ml.ninux.org/mailman/listinfo/ninux-dev + +Changelog +--------- + +See `CHANGES `_. + +License +------- + +See `LICENSE `_. + +This projects bundles third-party javascript libraries in its source code: + +- `D3.js (BSD-3-Clause) `_