updated cassandra docs

Author: voith

docs/source/topics/frontera-settings.rst

@@ -492,63 +492,56 @@ documents scheduled after the change. All previously queued documents will be cr
 Cassandra
 ---------
 
+.. setting:: CASSANDRABACKEND_CACHE_SIZE
 
-.. setting:: CASSANDRABACKEND_DROP_ALL_TABLES
+CASSANDRABACKEND_CACHE_SIZE
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-CASSANDRABACKEND_DROP_ALL_TABLES
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Default:: ``10000``
 
-Default: ``False``
+Cassandra Metadata LRU Cache size. It's used for caching objects, which are requested from DB every time already known,
+documents are crawled. This is mainly saves DB throughput, increase it if you're experiencing problems with too high
+volume of SELECT's to Metadata table, or decrease if you need to save memory.
 
-Set to ``True`` if you need to drop of all DB tables on backend instantiation (e.g. every Scrapy spider run).
 
-.. setting:: SQLALCHEMYBACKEND_ENGINE
+.. setting:: CASSANDRABACKEND_CLUSTER_HOSTS
 
-CASSANDRABACKEND_CLUSTER_IPS
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+CASSANDRABACKEND_CLUSTER_HOSTS
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Default:: ``['127.0.0.1']``
 
-Set IPs from Cassandra Cluster. Default is localhost. To assign more than one IP use this Syntax: ``['192.168.0.1', '192.168.0.2']``
+The list of contact points to try connecting for cluster discovery. All contact points are not required, the driver
+discovers the rest.
+
+.. setting:: CASSANDRABACKEND_CLUSTER_PORT
 
 CASSANDRABACKEND_CLUSTER_PORT
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Default:: ``9042``
 
-Set port from Cassandra Cluster / Nodes
+The server-side port to open connections to Cassandra.
 
+.. setting:: CASSANDRABACKEND_DROP_ALL_TABLES
 
-CASSANDRABACKEND_GENERATE_STATS
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+CASSANDRABACKEND_DROP_ALL_TABLES
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Default:: ``False``
+Default: ``False``
 
-Set this to true if you want to create an extra Table for stats collection. In this table there will be pages crawled, links queued etv. counted up.
+Set to ``True``  to drop and create all DB tables on backend instantiation.
 
+.. setting:: CASSANDRABACKEND_KEYSPACE
 
 CASSANDRABACKEND_KEYSPACE
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Default:: ``frontera``
-
-Set cassandra Keyspace
-
-CASSANDRABACKEND_CREATE_KEYSPACE_IF_NOT_EXISTS
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Default:: ``True``
+Default:: ``crawler``
 
-Creates Keyspace if it not exist. Set to false if you frontera shouldn't check on every startup.
-
-
-CASSANDRABACKEND_CRAWL_ID
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Default:: ``default``
-
-Sets an ID in each table for the actual crawl. If you want to run another crawl from begining in same Table set to another Crawl ID. Its an Text field.
+Set Cassandra Keyspace.
 
+.. setting:: CASSANDRABACKEND_MODELS
 
 CASSANDRABACKEND_MODELS
 ^^^^^^^^^^^^^^^^^^^^^^^
@@ -559,11 +552,19 @@ Default::
         'MetadataModel': 'frontera.contrib.backends.cassandra.models.MetadataModel',
         'StateModel': 'frontera.contrib.backends.cassandra.models.StateModel',
         'QueueModel': 'frontera.contrib.backends.cassandra.models.QueueModel',
-        'CrawlStatsModel': 'frontera.contrib.backends.cassandra.models.CrawlStatsModel'
+        'FifoOrLIfoQueueModel': 'frontera.contrib.backends.cassandra.models.FifoOrLIfoQueueModel',
     }
 
-This is mapping with Cassandra models used by backends. It is mainly used for customization.
+This is mapping of Cassandra models used by backends. It is mainly used for customization.
+
+.. setting:: CASSANDRABACKEND_REQUEST_TIMEOUT
+
+CASSANDRABACKEND_REQUEST_TIMEOUT
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Default:: ``60``
 
+Timeout in seconds for every request made by the Cassandra driver for to Cassandra.
 
 Revisiting backend
 ------------------

docs/source/topics/frontier-backends.rst

@@ -254,33 +254,15 @@ For a complete list of all settings used for SQLAlchemy backends check the :doc:
     SQLAlchemy :class:`Backend <frontera.core.components.Backend>` implementation of a random selection
     algorithm.
 
-
-Revisiting backend
-^^^^^^^^^^^^^^^^^^
-
-Based on custom SQLAlchemy backend, and queue. Crawling starts with seeds. After seeds are crawled, every new
-document will be scheduled for immediate crawling. On fetching every new document will be scheduled for recrawling
-after fixed interval set by :setting:`SQLALCHEMYBACKEND_REVISIT_INTERVAL`.
-
-Current implementation of revisiting backend has no prioritization. During long term runs spider could go idle, because
-there are no documents available for crawling, but there are documents waiting for their scheduled revisit time.
-
-
-.. class:: frontera.contrib.backends.sqlalchemy.revisiting.Backend
-
-    Base class for SQLAlchemy :class:`Backend <frontera.core.components.Backend>` implementation of revisiting back-end.
-
 .. _frontier-backends-cassandra:
 
 Cassandra backends
 ^^^^^^^^^^^^^^^^^^
 
-This set of :class:`Backend <frontera.core.components.Backend>` objects will use `Cassandra`_ as storage for
+This set of :class:`Backend <frontera.core.components.Backend>` objects will use Cassandra as storage for
 :ref:`basic algorithms <frontier-backends-basic-algorithms>`.
 
-Cassandra is a NoSQL Colum-Store Database with Linear scalability and a SQL-Like Query Language.
-
-If you need to use your own `declarative cassandra models`_, you can do it by using the
+If you need to use your own  `cassandra models`_, you can do it by using the
 :setting:`CASSANDRABACKEND_MODELS` setting.
 
 This setting uses a dictionary where ``key`` represents the name of the model to define and ``value`` the model to use.
@@ -290,13 +272,46 @@ For a complete list of all settings used for Cassandra backends check the :doc:`
 .. class:: frontera.contrib.backends.cassandra.BASE
 
     Base class for Cassandra :class:`Backend <frontera.core.components.Backend>` objects.
-    It runs cassandra in multi-spider one worker mode with the FIFO algorithm.
+
+.. class:: frontera.contrib.backends.cassandra.FIFO
+
+    Cassandra :class:`Backend <frontera.core.components.Backend>` implementation of `FIFO`_ algorithm.
+
+.. class:: frontera.contrib.backends.cassandra.LIFO
+
+    Cassandra :class:`Backend <frontera.core.components.Backend>` implementation of `LIFO`_ algorithm.
+
+.. class:: frontera.contrib.backends.cassandra.BFS
+
+    Cassandra :class:`Backend <frontera.core.components.Backend>` implementation of `BFS`_ algorithm.
+
+.. class:: frontera.contrib.backends.cassandra.DFS
+
+    Cassandra :class:`Backend <frontera.core.components.Backend>` implementation of `DFS`_ algorithm.
 
 .. class:: frontera.contrib.backends.cassandra.Distributed
 
-    Cassandra :class:`Backend <frontera.core.components.Backend>` implementation of the distributed Backend.
+    Cassandra :class:`Backend <frontera.core.components.Distributed>` implementation of a distributed backend.
+
+Revisiting backend
+^^^^^^^^^^^^^^^^^^
+
+There are two backends for Revisiting which are based on Cassandra and SqlAlchemy Backend and Queue. Crawling starts
+with seeds. After seeds are crawled, every new document will be scheduled for immediate crawling. On fetching every new
+document will be scheduled for recrawling after fixed interval set by :setting:`SQLALCHEMYBACKEND_REVISIT_INTERVAL` or
+:setting:`CASSANDRABACKEND_REVISIT_INTERVAL`.
+
+Current implementation of revisiting backend has no prioritization. During long term runs spider could go idle, because
+there are no documents available for crawling, but there are documents waiting for their scheduled revisit time.
+
+
+.. class:: frontera.contrib.backends.sqlalchemy.revisiting.Backend
+
+    Base class for SQLAlchemy :class:`Backend <frontera.core.components.Backend>` implementation of revisiting back-end.
 
+.. class:: frontera.contrib.backends.cassandra.revisiting.Backend
 
+    Base class for Cassandra :class:`Backend <frontera.core.components.Backend>` implementation of revisiting back-end.
 
 HBase backend
 ^^^^^^^^^^^^^
@@ -325,3 +340,4 @@ setting.
 .. _SQLAlchemy: http://www.sqlalchemy.org/
 .. _any databases supported by SQLAlchemy: http://docs.sqlalchemy.org/en/latest/dialects/index.html
 .. _declarative sqlalchemy models: http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/index.html
+.. _cassandra models: https://datastax.github.io/python-driver/cqlengine/models.html

frontera/settings/default_settings.py

@@ -10,18 +10,17 @@
 
 CANONICAL_SOLVER = 'frontera.contrib.canonicalsolvers.Basic'
 CASSANDRABACKEND_CACHE_SIZE = 10000
+CASSANDRABACKEND_CLUSTER_HOSTS = ['127.0.0.1']
+CASSANDRABACKEND_CLUSTER_PORT = 9042
 CASSANDRABACKEND_DROP_ALL_TABLES = False
+CASSANDRABACKEND_KEYSPACE = 'crawler'
 CASSANDRABACKEND_MODELS = {
     'MetadataModel': 'frontera.contrib.backends.cassandra.models.MetadataModel',
     'StateModel': 'frontera.contrib.backends.cassandra.models.StateModel',
     'QueueModel': 'frontera.contrib.backends.cassandra.models.QueueModel',
     'FifoOrLIfoQueueModel': 'frontera.contrib.backends.cassandra.models.FifoOrLIfoQueueModel',
 }
-CASSANDRABACKEND_REVISIT_INTERVAL = timedelta(days=1)
-CASSANDRABACKEND_CLUSTER_HOSTS = ['127.0.0.1']
-CASSANDRABACKEND_CLUSTER_PORT = 9042
-CASSANDRABACKEND_KEYSPACE = 'crawler'
-CASSANDRABACKEND_REQUEST_TIMEOUT = 100
+CASSANDRABACKEND_REQUEST_TIMEOUT = 60
 CASSANDRABACKEND_REVISIT_INTERVAL = timedelta(days=1)
 
 DELAY_ON_EMPTY = 5.0