Share and Money Management  0.16
Public Member Functions | Public Attributes | Static Public Attributes | Private Member Functions | Private Attributes | Static Private Attributes | List of all members
smm.quote.base.FetchPluginBase Class Reference

Base class of all quotation classes. More...

Public Member Functions

def __init__
 Constructor.
def getCurrent
 Fetch quotes of all shares from the data provider.
def getSoup
 Cleanup the HTML code and create a BeautifulSoup.BeautifulSoup instance.
def getNextTagText
 Extract the content (text) of the next element.
def initShareStruct
 Return an empty initialised share dict.
def getHTML
 Query a remote HTML page.
def getRemotePage
 Return the content of a remote page.
def getConnection
 Return a connection object to a remote page.
def getStringFromHtml
 Search for a text pattern in a BeautifulSoup.BeautifulSoup instance.
def timestamp2datetime
 Convert a timestamp string into a datetime object.
def toDecimal
 Convert a unicode string into a decimal value.
def attr2decimal
 Converts an attribute into a decimal value.

Public Attributes

 properties
 logger
 Instance logger.

Static Public Attributes

list properties = []
 List of supported properties.
string encoding = 'ascii'
 Default encoding of all returned strings.
int interface_version = 2
 Version number of the interface to access and convert the data.
string logger_name = 'smm.quote.base'
 Name of the instance logger.
string name = 'Override in derivated class!'
 Name of the data source.
 provider_id = None
 Unique provider ID.
string dsep = u','
 Decimal separator for converting strings to decimal values.
 tsep = None
 Thousands separator for converting strings to decimal values.
int max_retries = 3
 Number of retrys for temporary connection errors.
string user_agent = 'Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'
 Define a own user agent string.
tuple REC_REMOVE
 Pre compiled RE to remove useless data from the results.
string REP_EXCHANGE = r'(?P<ex_symbol>[A-Z]{1,3})'
 RE pattern to search / match exchange symbols.
string REP_ISIN = r''
 RE pattern to search / match ISIN.
string REP_SYMBOL = r'(?P<symbol>[A-Z0-9]{1,4})'
 RE pattern to search / match ticker symbols.
string REP_WKN = r'(?P<wkn>[a-zA-Z0-9]{6})'
 RE pattern to search / match German WKN.
int wait = 60
 Time to wait beween two retries in seconds.

Private Member Functions

def _getName
 Return a share name based on the share details.

Private Attributes

 _robots_cache
 Cache for already read and parsed robots.txt files.

Static Private Attributes

list _specific_properties = []
 Property names.
list _general_properties = ['errormsg', 'success']
 Name of the general properties.

Detailed Description

Base class of all quotation classes.

This class provides often used functions e.g. to access remote resources or to convert date and time from strings into Python objects.

Constructor & Destructor Documentation

def smm.quote.base.FetchPluginBase.__init__ (   self)

Constructor.

Member Function Documentation

def smm.quote.base.FetchPluginBase._getName (   self,
  share 
)
private

Return a share name based on the share details.

Parameters
selfThe object instance itself
shareDictionary with the share information

References smm.quote.base.toUnicode().

Referenced by smm.quote.yahoo.YahooCSVFetchPlugin.getCurrent(), smm.quote.yahoo.YahooWebFetchPlugin.getCurrent(), and smm.quote.base.FetchPluginBase.timestamp2datetime().

Here is the call graph for this function:

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.attr2decimal (   self,
  struct,
  attr,
  value 
)

Converts an attribute into a decimal value.

Errors will be catched smoothly. struck[attr] is deleted in case of errors.

Parameters
selfThe object instance itself
structShare dict
attrAttribute to store decimal value
valueUnicode string or BeautifulSoup.NavigableString, BeautifulSoup.Tag to convert into decimal value

References smm.quote.base.FetchPluginBase.dsep, smm.quote.base.FetchPluginBase.toDecimal(), smm.quote.base.toUnicode(), and smm.quote.base.FetchPluginBase.tsep.

Referenced by smm.quote.onvista.OnvistaFetchPlugin._extractCertificateDetails(), smm.quote.onvista.OnvistaFetchPlugin._extractFondDetails(), smm.quote.raiffeisen.RaiffeisenFetchPlugin.getCurrent(), smm.quote.vwd.VwdFetchPlugin.getCurrent(), smm.quote.yahoo.YahooCSVFetchPlugin.getCurrent(), and smm.quote.yahoo.YahooWebFetchPlugin.getCurrent().

Here is the call graph for this function:

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.getConnection (   self,
  url 
)

Return a connection object to a remote page.

The function opens a connection to a remote page and returns the the connection object or None. A explanatory messages is returned in case of errors.

The request will be repeated on temporary errors (socket.EAI_AGAIN). wait and max_retries controls this behaviour.

Function checks robots.txt and respect them.

Parameters
selfThe object instance itself
urlURL to open
Returns
A tuple of connection object and error message
See Also
getRemotePage()

References smm.quote.base.FetchPluginBase._robots_cache, smm.quote.base.FetchPluginBase.max_retries, smm.quote.base.toUnicode(), smm.quote.base.FetchPluginBase.user_agent, and smm.quote.base.FetchPluginBase.wait.

Referenced by smm.quote.onvista.OnvistaFetchPlugin.getCurrent(), smm.quote.base.FetchPluginBase.getRemotePage(), and smm.quote.onvista.OnvistaFetchPlugin.searchURL().

Here is the call graph for this function:

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.getCurrent (   self,
  share_list 
)

Fetch quotes of all shares from the data provider.

Parameters
selfThe object instance itself
share_listList of share symbols
Returns
A dictionary structure with the share name as key to a dictionary with all queried properties. The returned properties are strings.
See Also
properties
def smm.quote.base.FetchPluginBase.getHTML (   self,
  struct,
  key,
  url,
  pattern 
)

Query a remote HTML page.

Initialise the share specific structure using key (see initShareStruct()) and query a HTML page from the given url.

Querying is successful if pattern is found in the text.

In case of error contains struct[key]['errormsg'] a detailed error message.

Parameters
selfThe object instance itself
structShare dict
keyKey to store share related values in struct
urlURL to open
patternSinple text pattern
Returns
HTML as string or None
See Also
initShareStruct()

References smm.quote.base.FetchPluginBase.getRemotePage(), and smm.quote.base.FetchPluginBase.initShareStruct().

Referenced by smm.quote.raiffeisen.RaiffeisenFetchPlugin.getCurrent(), and smm.quote.yahoo.YahooWebFetchPlugin.getCurrent().

Here is the call graph for this function:

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.getNextTagText (   self,
  NavString,
  trim = False 
)

Extract the content (text) of the next element.

The given parameter has to be a instance of BeautifulSoup.NavigableString and the next element has to be a instance of BeautifulSoup.Tag.

Parameters
selfThe object instance itself
NavStringBeautifulSoup.NavigableString
trimCut off unused characters / sequences (see REC_REMOVE)
Returns
Text of the next element or None in error case

References smm.quote.base.toUnicode().

Referenced by smm.quote.onvista.OnvistaFetchPlugin._extractCertificateDetails(), and smm.quote.onvista.OnvistaFetchPlugin.getCurrent().

Here is the call graph for this function:

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.getRemotePage (   self,
  url 
)

Return the content of a remote page.

The function fetchs a remote page and returns the content as a string. The content is None in case of errors and a explanatory messages is returned.

Parameters
selfThe object instance itself
urlURL to open
Returns
A tuple of page content and error message
See Also
getConnection()

References smm.quote.base.FetchPluginBase.getConnection().

Referenced by smm.quote.vwd.VwdFetchPlugin.getCurrent(), smm.quote.onvista.OnvistaFetchPlugin.getCurrent(), smm.quote.yahoo.YahooCSVFetchPlugin.getCurrent(), and smm.quote.base.FetchPluginBase.getHTML().

Here is the call graph for this function:

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.getSoup (   self,
  html 
)

Cleanup the HTML code and create a BeautifulSoup.BeautifulSoup instance.

Parameters
selfThe object instance itself
htmlHTML code as String
Returns
A BeautifulSoup.BeautifulSoup instance

Referenced by smm.quote.raiffeisen.RaiffeisenFetchPlugin.getCurrent(), smm.quote.vwd.VwdFetchPlugin.getCurrent(), smm.quote.onvista.OnvistaFetchPlugin.getCurrent(), and smm.quote.yahoo.YahooWebFetchPlugin.getCurrent().

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.getStringFromHtml (   self,
  soup,
  text 
)

Search for a text pattern in a BeautifulSoup.BeautifulSoup instance.

Parameters
selfThe object instance itself
soupA BeautifulSoup.BeautifulSoup instance
textPattern to search for
Returns
The next string element follow to the text pattern or None

Referenced by smm.quote.yahoo.YahooWebFetchPlugin.getCurrent().

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.initShareStruct (   self)

Return an empty initialised share dict.

Parameters
selfThe object instance itself

References smm.quote.base.FetchPluginBase.name, smm.objects.Share.name, and smm.quote.base.FetchPluginBase.provider_id.

Referenced by smm.quote.vwd.VwdFetchPlugin.getCurrent(), smm.quote.onvista.OnvistaFetchPlugin.getCurrent(), smm.quote.yahoo.YahooCSVFetchPlugin.getCurrent(), smm.quote.yahoo.YahooWebFetchPlugin.getCurrent(), and smm.quote.base.FetchPluginBase.getHTML().

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.timestamp2datetime (   self,
  share,
  timestamp,
  fmt,
  loc = None 
)

Convert a timestamp string into a datetime object.

Unicode strings for format and date / time information are converted into plain ascii automatically.

Parameters
selfThe object instance itself
shareDictionary with the share information
timestampFormat of the date string
fmtFormat of the timestamp string
locLocale used for decode timestamps, default 'C'

On success share['success'] is set to True and share['datetime'] contains the datetime object.

On failure share['success'] will set to False and a detailed error message will stored in share['errormsg']. share['datetime'] isn't defined.

Note
A ValueError exception is rasing during conversion of the am / pm suffix because some locale don't know those suffixes. We switch to 'C' locale if the decoding directive for am / pm %p is found in the format string.

References smm.quote.base.FetchPluginBase._getName(), and smm.quote.base.toUnicode().

Referenced by smm.quote.onvista.OnvistaFetchPlugin._extractCertificateDetails(), smm.quote.onvista.OnvistaFetchPlugin._extractFondDetails(), smm.quote.raiffeisen.RaiffeisenFetchPlugin.getCurrent(), smm.quote.vwd.VwdFetchPlugin.getCurrent(), smm.quote.yahoo.YahooCSVFetchPlugin.getCurrent(), and smm.quote.yahoo.YahooWebFetchPlugin.getCurrent().

Here is the call graph for this function:

Here is the caller graph for this function:

def smm.quote.base.FetchPluginBase.toDecimal (   self,
  value,
  dsep = u',
  tsep = u'' 
)

Convert a unicode string into a decimal value.

Parameters
selfThe object instance itself
valueUnicode string to convert into decimal value
dsepDecimal separator
tsepThousands separator
Returns
Decimal valued

Leading and tailing spaces will be removed automatically.

The percent sign is cut off and the value is divided by 100. 100% is Decimal("1").

Note
None will not processed

Examples:

>>> self.toDecimal(u'12%')
Decimal("0.12")
>>> self.toDecimal(u'12 %')
Decimal("0.12")
>>> self.toDecimal(u'12,30')
Decimal("12.30")
>>> self.toDecimal(u'12.30')
Decimal("12.30")
>>> self.toDecimal(u'12.000', tsep=u'.')
Decimal("12000")
>>> self.toDecimal(u'12.000%', tsep=u'.')
Decimal("120")
Exceptions
ValueErrorRaised for not convertible values
TypeErrorRaised for non unicode value

References smm.quote.base.toUnicode().

Referenced by smm.quote.base.FetchPluginBase.attr2decimal().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

list smm.quote.base.FetchPluginBase._general_properties = ['errormsg', 'success']
staticprivate

Name of the general properties.

smm.quote.base.FetchPluginBase._robots_cache
private

Cache for already read and parsed robots.txt files.

Referenced by smm.quote.base.FetchPluginBase.getConnection().

list smm.quote.base.FetchPluginBase._specific_properties = []
staticprivate

Property names.

string smm.quote.base.FetchPluginBase.dsep = u','
static

Decimal separator for converting strings to decimal values.

See Also
attr2decimal()
tsep

Referenced by smm.quote.base.FetchPluginBase.attr2decimal().

string smm.quote.base.FetchPluginBase.encoding = 'ascii'
static

Default encoding of all returned strings.

Referenced by gui.log.FileHandler._open(), and gui.log.StringHandler._toUnicode().

int smm.quote.base.FetchPluginBase.interface_version = 2
static

Version number of the interface to access and convert the data.

smm.quote.base.FetchPluginBase.logger

Instance logger.

string smm.quote.base.FetchPluginBase.logger_name = 'smm.quote.base'
static

Name of the instance logger.

int smm.quote.base.FetchPluginBase.max_retries = 3
static

Number of retrys for temporary connection errors.

See Also
wait

Referenced by smm.quote.base.FetchPluginBase.getConnection().

string smm.quote.base.FetchPluginBase.name = 'Override in derivated class!'
static

Name of the data source.

Referenced by smm.quote.base.FetchPluginBase.initShareStruct().

list smm.quote.base.FetchPluginBase.properties = []
static

List of supported properties.

Every class should provide a minimum set of properties

  • date Time tuple with the date and time (if available) of the quote
  • errormsg As unicode string or None
  • quote.price As ascii string
  • source Source of the data as unicode string
  • success Flag as boolean value
smm.quote.base.FetchPluginBase.properties
smm.quote.base.FetchPluginBase.provider_id = None
static

Unique provider ID.

Referenced by smm.quote.base.FetchPluginBase.initShareStruct().

tuple smm.quote.base.FetchPluginBase.REC_REMOVE
static
Initial value:
re.compile(
r'(&nbsp;)' # protected whitespaces
r'|'
r'(&plusmn;)' # plus minus
r'|'
r'(\.)' # thousands separator "."
r'|'
r'(\%?\s*$)' # tailing %
r'|'
r'(\s*\+?)' # + sign'
)

Pre compiled RE to remove useless data from the results.

string smm.quote.base.FetchPluginBase.REP_EXCHANGE = r'(?P<ex_symbol>[A-Z]{1,3})'
static

RE pattern to search / match exchange symbols.

string smm.quote.base.FetchPluginBase.REP_ISIN = r''
static

RE pattern to search / match ISIN.

Note
Compile with re.VERBOSE
string smm.quote.base.FetchPluginBase.REP_SYMBOL = r'(?P<symbol>[A-Z0-9]{1,4})'
static

RE pattern to search / match ticker symbols.

string smm.quote.base.FetchPluginBase.REP_WKN = r'(?P<wkn>[a-zA-Z0-9]{6})'
static

RE pattern to search / match German WKN.

smm.quote.base.FetchPluginBase.tsep = None
static

Thousands separator for converting strings to decimal values.

See Also
attr2decimal()
dsep

Referenced by smm.quote.base.FetchPluginBase.attr2decimal().

string smm.quote.base.FetchPluginBase.user_agent = 'Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'
static

Define a own user agent string.

Set to None for disable changing user agent

Referenced by smm.quote.base.FetchPluginBase.getConnection().

int smm.quote.base.FetchPluginBase.wait = 60
static

Time to wait beween two retries in seconds.

See Also
max_retries

Referenced by smm.quote.base.FetchPluginBase.getConnection().


The documentation for this class was generated from the following file: