PISCES Code Reference

PISCES Package

Please note that ONLY the functions in the “api” module are officially supported.

api Module

PISCES.api.connect_orm()
PISCES.api.distinct(expr)

Produce an column-expression-level unary DISTINCT clause.

This applies the DISTINCT keyword to an individual column expression, and is typically contained within an aggregate function, as in:

from sqlalchemy import distinct, func
stmt = select([func.count(distinct(users_table.c.name))])

The above would produce an expression resembling:

SELECT COUNT(DISTINCT name) FROM user

The distinct() function is also available as a column-level method, e.g. ColumnElement.distinct(), as in:

stmt = select([func.count(users_table.c.name.distinct())])

The distinct() operator is different from the Select.distinct() method of Select, which produces a SELECT statement with DISTINCT applied to the result set as a whole, e.g. a SELECT DISTINCT expression. See that method for further information.

See also

ColumnElement.distinct()

Select.distinct()

func

PISCES.api.get_common_names_in_hucs(huc_list)

Given a list of HUC12 IDs, returns a list of species common names present (for all presence types) in those HUC12s :param huc_list: list: A list of HUC 12 ids to search :return: list: Common names of species in the huc12s provided

PISCES.api.get_observation_records_for_hucs(huc_list_or_layer, species_list=None, presence_types_list=None, collection_list=None)
param huc_list_or_layer:
 
param species_list:
 a Python list of PISCES species codes
param presence_types_list:
 a Python list of presence type codes (from definitions table) to limit the results to
param collection_list:
 a Python list of collection names to limit the results to

Given a set of hucs as a list or a feature class or feature layer, and one or more species or groups, returns the observation records meeting those criteria

Returns:list of orm.Observation objects
PISCES.api.get_query_as_layer(query, bind=None, fc=False, callback=None, callback_args=None, metadata=False)
Takes a SQL query that obtains HUC_12 IDs in the PISCES database and returns a feature class or feature layer generated from that query. Contains the full PISCES metadata available.
Parameters:
  • query – str: The query to run against the PISCES database that will generate a set of HUC IDs. It must conform to PISCES query standards (namely, that the HUC_12 field must come out as Zone_ID to bne seen). This may change in a future version. You should read the rest of the documentation before using this function.
  • bind – A bind value to pass along with the query. If you need more than one, use {bind}
  • fc – boolean. Indicates whether to return a feature class or a feature layer. Feature layer is the default (fc is False). When fc is True, a feature class is written out and the path is returned instead of a feature layer.
  • callback
  • callback_args
Returns:

Fully qualified path to generated feature class on disk

callbacks Module

PISCES.callbacks.add_field(zones_layer, db_cursor, cb_args, parent_layer)
Very similar to postprocess_zones except it executes a query once, and joins a result to the output layer Meant to be used for summary queries that summarize by HUC12
Parameters:
  • zones_layer
  • db_cursor
  • cb_args
  • parent_layer
Returns:

PISCES.callbacks.clipped_layer_overlay(zones_layer, db_cursor, cb_args, parent_layer)
PISCES.callbacks.coerce_value_from_arc_type(val, col_type)
PISCES.callbacks.connectivity(zones_layer, db_cursor, args, map_layer)
PISCES.callbacks.connectivity_recurse(zones_layer, huc, select_vars, atts, huc_cursor)
PISCES.callbacks.diversity(zones_layer, db_cursor, args, map_layer)

Generates 4 different layers based upon args args = native/non,dams/no,divtype,qc/noqc

PISCES.callbacks.diversity_assemblage_to_cname(assemblage)
PISCES.callbacks.diversity_count(current_watershed, all_watersheds)
PISCES.callbacks.diversity_difference()
PISCES.callbacks.diversity_jaccard(current_watershed, all_watersheds)

returns Jaccard DISSIMILARITY

class PISCES.callbacks.empty_object

@todo: this should be replaced in usage with object()

PISCES.callbacks.export_raster(zones_layer, db_cursor, args, parent_layer)
Parameters:
  • zones_layer
  • db_cursor
  • args
  • parent_layer
Returns:

Callback args are 1: value code - the value to assign presence in the output raster 2: template raster - what raster to use for cell size, extent, analysis mask, and snap raster 3: database - what database to export the raster to (FGDB, PGDB, folder). This is not technically necessary because the raster will get saved in the layer cache with the maps, but if you want a dump of the rasters elsewhere, use this 4: file extension to export to in database - include period where necessary. For GDBs, leave as None or empty

PISCES.callbacks.find_upstream(watershed, all_watersheds, dams_flag=False)
PISCES.callbacks.forest_listing(zones_layer, db_cursor, args)
class PISCES.callbacks.function_arg(function, argument)

a small class that allows us to pass around function names with an argument (or set of arguments) in a way that lets us test for it

PISCES.callbacks.genus_family_richness(zones_layer, db_cursor, gf, parent_layer)
DEPRECATED

@param zones_layer: @param db_cursor: @param gf: @param parent_layer: @return:

PISCES.callbacks.get_arg(args, index, default, fieldmap=None, split_commas=False)
PISCES.callbacks.get_diversity_into_network(dams_flag, db_cursor, native_flag, qc_flag, upstream_flag, zones_layer)
PISCES.callbacks.get_downstream_diversities(t_layer, db_cursor, cb_args, parent_layer)
Standard callback - takes a layer of hucs, gets the assemblage for all hucs, then adds two fields with that data 1) contains the downstream assemblage and 2) contains the count of downstream taxa

@param t_layer: @param db_cursor: @param cb_args: @param parent_layer: @return: t_layer

PISCES.callbacks.make_tooltip_column(zones_layer, db_cursor, args)
PISCES.callbacks.mega_diversity_info(zones_layer, db_cursor, args, parent_layer)

runs a large number of diversity functions on a single layer and returns it. These functions all add columns with data to one layer!

Metadata: In order

Note that since this function runs a number of other mapping functions, there will be some data duplication. Many of the non-qced output columns can now be deprecated since all data of interest is currently qced. They are here for legacy purposes and for any future changes

——– BASIC RICHNESS ——– native_richness: integer. Raw alpha richness as count of number of native fish species in the HUC across PISCES. native_qc_richness: integer. Same as above, but only for species in a data collection that is qced native_qc_historic_richness: integer. Same as native_qc_richness, but only for historic data types

nonnative_richness: integer. Alpha richness of nonnatives. As of this writing, very little data on this. nonnative_qc_richness: integer. same as previous, but for data in qc dataset. Currently none for nonnatives, but here for future-proofing.

all_richness: integer. Alpha richness of all fish species (native or nonnative) in PISCES, does not include amphibs and reptiles, etc. all_qc_richness: integer. Same as above, but for qced data. Should match native_qc_richness as of this writing.

sens_nat_rich: integer. Richness of sensitive native species. Alpha richness of native fish taxa in PISCES where the Moyle 2011 average score <= 3 and >= 0. Uses the field Species_Aux.Average in PISCES for the score. sens_nat_qc_rich: integer. Same as previous except only for qced data in each HUC12

sens_nat_assem: Assemblage of sensitive species in the HUC12 where sensitive is defined as in sens_nat_rich sens_nat_qc_assem: same as sens_nat_assem, but only for qc taxa

——– SENSITIVITY STATS ——– avg_sensitivity_score: real (decimal). The average score (Species_Aux.Average) for all taxa in the HUC12 avg_sensitivity_under3: real (decimal). same as above, but only including species with scores <= 3 and >= 0 avg_sensitivity_under2: real (decimal). same as above but <=2 and >=0 avg_sensitivity_under1: real (decimal). same as above but <=1 and >=0

avg_qc_sensitivity_score: real (decimal). The average score (Species_Aux.Average) for all QCed taxa in the HUC12 avg_qc_sensitivity_under3: real (decimal). same but <=3 and >=0 avg_qc_sensitivity_under2: real (decimal). same but <=2 and >=0 avg_qc_sensitivity_under1: real (decimal). same but <=1 and >=0

——– LOSSES AND EXTIRPATIONS ——– All data in this section is related to native fish and qced data

current_richness: integer. QCed alpha richness for native fish species. Includes translocations current_assemblage: Assemblage of qced native fish species in the HUC12. Includes translocations

historic_richness: integer. QCed historic alpha richness for native fish species historic_assemblage: Assemblage of qced native fish species historically in the HUC12

losses: integer. Effectively, historic_richness - current_richess, except excludes translocations from current. Essentially, a count of extirpated species losses_list: The assemblage of species extirpated from that HUC12. gains: integer. How many native fish taxa have been translocated into that HUC12? gains_list. The assemblage of translocated species

richness_difference: integer. count of the number of species gained-lost (so # currently - # historically). When the layer is exported without translocations included, this # may not correctly include translocations.

——– JACCARD DISTANCES ——– To be updated later. Please ignore these for now

——– DOWNSTREAM INFO ——– downstream_assemblage: Assemblage of species in the HUC12 downstream

@param zones_layer: @param db_cursor: @param args: @param parent_layer: @return:

PISCES.callbacks.network_distance(zones_layer, db_cursor, args, map_layer)
PISCES.callbacks.network_get_distance(zone1, zone2, zones_info)
PISCES.callbacks.network_get_path(zone, zones_downstream, path_list)
PISCES.callbacks.postprocess_zones(zones_layer, db_cursor, cb_args, parent_layer)

Takes each record in the zones_layer and runs queries related to them. Queries should specify the place to include the zone with sql bind variables ”?” and the variable operators specified by object location relative to parent_layer (eg: {custom_query.bind_var}) are optional Flexible:{object path from parent layer being worked on} - example {custom_query.bind_var} would return parent_layer.custom_query.bind_var and dump it into the query at that location Argument Format: query,column_name,column_data_type,query,column_name,column_data_type,query...etc Notes: the value to be placed in the column for a given query should be selected “AS col_value” - that’s what the following code will be looking to retrieve

class PISCES.callbacks.query

An ORM would do many of the things that this class does, but we don’t have one, so this will have to work

add_base()
add_zonal_callback()
compose()
make_count_query()
make_into_qc()
set_current()
set_defaults(zonal_callback=True, qc=False)

Handles the proper method chaining for most queries

set_historic()
PISCES.callbacks.replace_variables(replacement_string, parent_layer)
PISCES.callbacks.representation(zones_layer, db_cursor, cb_args, parent_layer)

just an alias for clipped_layer_overlay

PISCES.callbacks.richness(zones_layer, db_cursor, cb_args, parent_layer)
DEPRECATED: An old richness function - superceded by passing the correct parameters to postprocess_zones

@param zones_layer: @param db_cursor: @param cb_args: @param parent_layer: @return:

PISCES.callbacks.richness_difference(zones_layer, db_cursor, args, parent_layer)
PISCES.callbacks.sensitivity_stats(zones_layer, db_cursor, args, parent_layer)
Parameters:
  • zones_layer
  • db_cursor
  • args
  • parent_layer
Returns:

PISCES.callbacks.split_postprocess_cbargs(cb_args, layer, parent_layer)
PISCES.callbacks.stringify(in_list, delimiter=', ')

Turns a list into a string without the “u” or brackets @param in_list: list to process @param delimiter: string denoting what to use to separate items @return: string

PISCES.callbacks.subprocess_assemblage(zones_layer, db_cursor, cb_args, parent_layer)

a subfunction for use with postprocess_zones. This entire function could be done in a single query, but Access won’t run it. So...here we are.

PISCES.callbacks.subprocess_losses(zones_layer, db_cursor, cb_args, parent_layer)

a subfunction for use with postprocess_zones. This entire function could be done in a single query, but Access won’t run it. So...here we are. #cb_args[0] contains the function_arg object #cb_args[1] contains the bind variable (the huc) #cb_args[2] contains all of the args to postprocess_zones

PISCES.callbacks.ted_temp(zones_layer, db_cursor, args, parent_layer)
PISCES.callbacks.test_callback(zones_layer, db_cursor, args, parent_layer)
Just a callback that prints out all of the arguments passed in, for testing purposes
Parameters:
  • zones_layer
  • db_cursor
  • args
  • parent_layer
Returns:

class PISCES.callbacks.watershed
PISCES.callbacks.write_results_by_key(layer, results, layer_field, results_field, layer_key='HUC_12', results_key='HUC_12')

comparison Module

PISCES.comparison.batch_compare_species_ranges(output_file, species=None)
Pulls the list of native species (since they have historic ranges) from the database and runs :function::historic_vs_current on each one. Returns a csv file with the results
Parameters:
  • output_file (str) – Path to new CSV file where results would be placed
  • species (list) – optional. An iterable of species codes to run this for
PISCES.comparison.centroid_distance(range_old, range_new)
PISCES.comparison.historic_vs_current(fid, collections='5, 15')
Compares a species’ historic range to its current range, defaulting to using only hq_collections
Parameters:
  • fid (str) – A PISCES species FID code (eg: “SOM09”)
  • collections (str) – the collection ids to limit the data to. Defaults to the hq_collections variable which uses the “QC dataset”
Returns:

comparison object

Return type:

geospatial.geometry.Comparison

PISCES.comparison.percent_overlap(range_old, range_new)

config Module

funcs Module

PISCES.funcs.change_config_item(regex, value, fiu)
Changes a single value in the config.py
Parameters:
  • regex
  • value
Returns:

PISCES.funcs.check_workspace()
Checks to make sure we have the temporary working locations. Creates them if we don’t. Errors get raised if not.
Returns:
PISCES.funcs.clean_location(path, path_type='FGDB')
Cleans and recreates a particular type of workspace. Types include file geodatabases, personal geodatabases, and folders
Parameters:
  • path – the full path to the item to clean
  • path_type – one of the following: (FGDB, PGDB, Folder) for File Geodatabase, Personal Geodatabase, and folders, respectively
Returns:

PISCES.funcs.clean_workspace(softerror=False)
PISCES.funcs.compose_query_from_dict(table, value_pairs)
PISCES.funcs.copy_data(feature_class, new_location)
A wrapper for copying data to gdbs that checks for existence and creates unique names if need be
Parameters:
  • feature_class
  • new_location
Returns:

PISCES.funcs.create_collection(name, short_name, description, db_cursor)
PISCES.funcs.data_stats(db_cursor, print_to_screen=True, name_col='Parameter', value_col='Value')
PISCES.funcs.db_close(cursor, conn)
PISCES.funcs.db_connect(db_name, note=None, access=False)
PISCES.funcs.fully_qualified_path(relative_item)

takes a path relative to the PISCES root and returns its full path. Somewhat trivial, but complicated when the fully PISCES code isn’t running. Does no validation of inputs to ensure it’s a true path

PISCES.funcs.get_last_insert_id(db_cursor, access=False)

A shortcut function to return the ID of the last insert operation. :param db_cursor: The pyodbc cursor that the insert occurred on. :return: integer ID of last record

PISCES.funcs.get_path()

Obtains the PISCES install path from the windows system registry - used for scripts running in ArcGIS where the current working directory isn’t the PISCES install path.

Returns:str base_folder
Raise:WindowsError on failure to retrive info from the registry
PISCES.funcs.get_species_from_alt_code(l_alt_code_species, filter_code)
PISCES.funcs.get_species_in_dataset(dataset_id, db_cursor)
PISCES.funcs.hucs_to_list(dataset)

takes an arcgis dataset containing a field with “HUC_12” in it and transforms the unique values in that field into a list

PISCES.funcs.import_new_data(dataset_name=None)
PISCES.funcs.is_between(check_num, num1, num2)
PISCES.funcs.match_species_from_string(species_string, validate=True, return_all=False)

Given a string, finds potential species codes within it - returning only the first match unless return_all is True. When validate is false, it returns whatever it finds, without checking to see if those codes exist. If validate is True, it will ensure all codes are valid first :param species_string: The string potentially containing species codes :param validate: when True (default), this function checks to ensure the species code exists. Otherwise, it assumes any pattern of three characters followed by two numbers is a species code. If validate is true, the data_setup function must have already been run (or equivalent) to populate local_vars.all_fish :param return_all: boolean. causes the function to return all matches. Returns a list. :return:

PISCES.funcs.new_data_dedupe()
PISCES.funcs.new_data_fetch(dataset_name=None)
PISCES.funcs.new_data_get_current_observation_sets(l_connection)
PISCES.funcs.new_data_pull(dataset_name=None)
PISCES.funcs.new_data_validate()
PISCES.funcs.query_to_list_of_dicts(query, db_cursor, drop_fields=None)
PISCES.funcs.refresh_cursor(database=None, connection=None, existing_cursor=None, close_connection=False)
PISCES.funcs.refresh_layer_cache(l_db_cursor=None)

deletes the existing layer cache and recreates it :param layer_cache_location: relative path to layer cache

PISCES.funcs.remove_data(item_name, l_workspace=u'C:\\Users\\dsx.AD3\\Code\\PISCES\\proc\\calculations.gdb')
PISCES.funcs.run_query(query)

A simple helper function that helps run a query with less fuss

PISCES.funcs.setup_new_data(obs_sets)
PISCES.funcs.setup_test_mode()
PISCES.funcs.species_group_as_list(group_name=None, short_name=None, db_cursor=None, warn=False)
PISCES.funcs.text_to_species_list(text_input, multispecies_first=True, warn=False)
TODO: Write Unit Tests! Given a string, figures out whether it’s a species or group and returns the species code. Accepts species codes, our ArcGIS tool format, and group names. If the string is a comma or semicolon separated list of such strings or groups, it splits and determines the species codes for all items involved.
Parameters:
  • text_input – The string to parse for species.
  • multispecies_first – indicates the order to run the splits and tests in. If you know you’re passing in a comma/semicolon separated list of species, leave this as True. If it’s a single, or you really know what you’re doing, you can set it to False
Returns:

input_filters Module

class PISCES.input_filters.CNDDB_IF(full_obs_set)

Bases: PISCES.input_filters.Gen_Table_IF

Subclasses Gen_Table_IF to override presence type handling to map CNDDB presence types to PISCES

handle_obs_type(item, observation, row, method)
class PISCES.input_filters.Filtered_Table_IF(full_obs_set)

Bases: PISCES.input_filters.Gen_Table_IF

A subclass of Gen_Table_IF designed to automatically mark new non-historical records that coincide with existing QC data as QCed (to reinforce that data), then leaving other data not-QCed, but available in observations.

filter_records()
process_observations()

Overrides parent class, but calls parent class version first, then calls the filter_records method

class PISCES.input_filters.Gen_Poly_IF(full_obs_set)

Bases: PISCES.input_filters.input_filter

determine_species()
process_observations()
class PISCES.input_filters.Gen_Table_IF(full_obs_set)

Bases: PISCES.input_filters.input_filter

Generic Filter for tables that have a species column and a column for x/y - we can base future classes on this subclasses can extend the functionality of this class by polymorphosing (defining a new) the method process_observations, and then calling parent class’s version of .process_observations() after all the prep work is done. This would be a good way to do things like splitting data columns and any other very item specific processing (generic processing would ideally) be incorporated into this class.

Even better than overriding process_observations is to leave that in place, and instead override the handler functions on this class (handle_zone_id, handle_date, etc). This allows the primary workflow of the class to remain, but provides a way to change the behavior for specific fields as necessary. You can define multiple new handler functions for the same attribute in a subclass and allow the PISCES operator to specify which should be used at import time (in the field map) in order to provide customization options for a specific type of dataset.

This class, or one of its subclasses, should be used for most imports as it is the most flexible and tested.

determine_species()
Overrides the default method for determining the species of an input filter (with respect to the whole dataset). Datasets have a species code and records have separate species code. This one determines the code for the whole dataset, using MUL99 when it deals with multiple species
Returns:
determine_species_for_record(species)

Receives the value for the species field for each record and attempts to look it up using the main species codes for PISCES or using the alt codes for the current input filter using this class.

This is a key method to override in a subclass handling data of a differing form or format.

Parameters:species
Returns:
get_handler(l_map)
Given a field map object, attaches the actual handler function to it for use during translation
Parameters:l_map
Returns:
get_zones()

Provided a table with coordinates or a feature class, returns a list with the spatially joined attributes of the input dataset on HUC12s, allowing for a translation of input record data to HUC12 locations

This function is a key function to override for data of differing form or formats.

Returns:list: full, spatially joined record information for all HUCs the dataset passed in had data for
handle_args()
Loads arguments provided in the newdata configuration and makes them available to the current instance
Returns:
handle_certainty(item, observation, row, method)

Certainty should be considered deprecated as it is not supported by the more modern PISCES tool meant to be overridden by subclasses - as it is, it just returns the default. This way, though, subclasses can override these small parts without having to override the entire process_observations() behemoth.

handle_date(item, observation, row, method)

This default method just returns the value of the field configured for dates, but you may wish to override it to convert dates to a consistent format

Override in a subclass for different behavior

Parameters:
  • item – The current parameter being processed - determined by the field map. Item is an instance of the field_map class
  • observation – an instance of the local_vars.observation class being populated by this handler. Think of this as the current state of the output rpow
  • row – The input data row (from the spatially joined HUC12 layer) containing HUC12 attributes and the input data
  • method – The IF_Method instance (for accessing other properties of the input filter, such as default values
Returns:

value to be used as date for the current record

handle_latitude(item, observation, row, method)

Basic function that returns the value of the latitude field

Override in a subclass for different behavior

Parameters:
  • item – The current parameter being processed - determined by the field map. Item is an instance of the field_map class
  • observation – an instance of the local_vars.observation class being populated by this handler. Think of this as the current state of the output rpow
  • row – The input data row (from the spatially joined HUC12 layer) containing HUC12 attributes and the input data
  • method – The IF_Method instance (for accessing other properties of the input filter, such as default values
Returns:

value to be used as latitude for the current record

handle_longitude(item, observation, row, method)

Basic function that returns the value of the longitude field

Override in a subclass for different behavior

Parameters:
  • item – The current parameter being processed - determined by the field map. Item is an instance of the field_map class
  • observation – an instance of the local_vars.observation class being populated by this handler. Think of this as the current state of the output rpow
  • row – The input data row (from the spatially joined HUC12 layer) containing HUC12 attributes and the input data
  • method – The IF_Method instance (for accessing other properties of the input filter, such as default values
Returns:

value to be used as longitude for the current record

handle_obs_type(item, observation, row, method)

By default a short decision tree, returning the value in the configured presence type field, or if that doesn’t exist, then the default presence type for the input filter

Override in a subclass for different behavior

Parameters:
  • item – The current parameter being processed - determined by the field map. Item is an instance of the field_map class
  • observation – an instance of the local_vars.observation class being populated by this handler. Think of this as the current state of the output rpow
  • row – The input data row (from the spatially joined HUC12 layer) containing HUC12 attributes and the input data
  • method – The IF_Method instance (for accessing other properties of the input filter, such as default values
Returns:

value to be used as observation (presence) type for the current record

handle_observer(item, observation, row, method)

Decision tree. If there’s an observer field, it returns that. Otherwise, if the parent dataset has an observer configured, it returns that. Finally, if we still don’t have an observer, it returns the input filter’s default observer

Override in a subclass for different behavior

Parameters:
  • item – The current parameter being processed - determined by the field map. Item is an instance of the field_map class
  • observation – an instance of the local_vars.observation class being populated by this handler. Think of this as the current state of the output rpow
  • row – The input data row (from the spatially joined HUC12 layer) containing HUC12 attributes and the input data
  • method – The IF_Method instance (for accessing other properties of the input filter, such as default values
Returns:

value to be used as observer for the current record

handle_species(item, observation, row, method)

Deprecated - use determine_species for record if you want to override TODO: this function should not actually get used - it’s a relic - consider removal!

Parameters:
  • item – The current parameter being processed - determined by the field map. Item is an instance of the field_map class
  • observation – an instance of the local_vars.observation class being populated by this handler. Think of this as the current state of the output rpow
  • row – The input data row (from the spatially joined HUC12 layer) containing HUC12 attributes and the input data
  • method – The IF_Method instance (for accessing other properties of the input filter, such as default values
Returns:

value to be used as species for the current record

handle_survey_method(item, observation, row, method)

By default a short decision tree, returning the value in the configured survey method field, or if that doesn’t exist, then the default survey method for the dataset

Override in a subclass for different behavior

Parameters:
  • item – The current parameter being processed - determined by the field map. Item is an instance of the field_map class
  • observation – an instance of the local_vars.observation class being populated by this handler. Think of this as the current state of the output rpow
  • row – The input data row (from the spatially joined HUC12 layer) containing HUC12 attributes and the input data
  • method – The IF_Method instance (for accessing other properties of the input filter, such as default values
Returns:

value to be used as the survey method for the current record

handle_zone_id(item, observation, row, method)

Default way to determine zone value for the current record. Since, by default, the Gen_Table_IF class spatially joins records, it just returns the value of the HUC12 field. Can be overridden by subclasses to behave differently though

Override in a subclass for different behavior

Parameters:
  • item – The current parameter being processed - determined by the field map. Item is an instance of the field_map class
  • observation – an instance of the local_vars.observation class being populated by this handler. Think of this as the current state of the output rpow
  • row – The input data row (from the spatially joined HUC12 layer) containing HUC12 attributes and the input data
  • method – The IF_Method instance (for accessing other properties of the input filter, such as default values
Returns:

value to be used as the zone/HUC12 for the current record

process_observations()

Does the bulk of the work for determining records. Calls get_zones to get the information out of the input table or feature class, but then does the translation to PISCES values.

Each item for tanslation is handled by a specific method, which can be overridden with a new code class. A code class can also have multiple methods for the same field and the field to use can be chosen by the person configuring the import in the field mapping area. Species and NotesItems are not handled by handlers directly, but Species is handled by the determine species for record method and can be overriden there

It is not recommended to override this method in a subclass. Instead, override the various handle_parameter methods

Returns:
class PISCES.input_filters.HUC12_IF(full_obs_set)

Bases: PISCES.input_filters.Gen_Table_IF

An input filter for datasets that have a HUC12 (Zone_ID) field that directly corresponds to our own HUC_12 layer

get_zones()
handle_zone_id(item, observation, row, method)
class PISCES.input_filters.Moyle_IF(full_obs_set)

Bases: PISCES.input_filters.Gen_Table_IF

determine_species()
determine_species_for_record(junk)
handle_certainty(item, observation, row, method)
class PISCES.input_filters.R5_Table_IF(full_obs_set)

Bases: PISCES.input_filters.Gen_Table_IF

define_presence_values()
handle_certainty(item, observation, row, method)
handle_obs_type(item, observation, row, method)
process_observations()

Rewrites the r5 tables so that species is a column with a record for each presence

class PISCES.input_filters.TNC_HUC12_IF(full_obs_set)

Bases: PISCES.input_filters.HUC12_IF

Nature conservancy specific overrides/field mapping

handle_obs_type(item, observation, row, method)
class PISCES.input_filters.TU_Inverts_IF(full_obs_set)

Bases: PISCES.input_filters.R5_Table_IF

Totally different datasets, but similar structure to the R5 tables. Columns are alt codes, rows are locations, species values coded for in the values

define_presence_values()
PISCES.input_filters.arcgis_table_to_list(dataset)
Might duplicate and shorten table_to_array
Parameters:dataset
Returns:
PISCES.input_filters.copy_row(new_row, old_row, fields, skip_fields)
PISCES.input_filters.determine_certainty(certainties)
class PISCES.input_filters.empty_field
class PISCES.input_filters.empty_row
fields_to_notes_string(fields)
PISCES.input_filters.feature_to_HUCs(feature=None, intersect_centroid='INTERSECT')

intersect_centroid is just whether we want to overlap by intersect or centroid methods

class PISCES.input_filters.field_map
PISCES.input_filters.find_HUC_by_latlong(lat, long)

provided a lat/long pair (in Teale Albers meters!), it returns the HUC

PISCES.input_filters.get_IF_list()
PISCES.input_filters.get_species_data()
class PISCES.input_filters.if_method
class PISCES.input_filters.input_filter(full_obs_set)
all_init(full_obs_set)
determine_species(match_string)
filter_results()

meant to provide a placeholder for functions that remove rows after processing and before inserting

get_defaults()
index_field_maps()
make_results(zones, species, obs_type, pri_cert, sec_cert=None, if_method=None)
process_observations()
set_up_field_maps()
PISCES.input_filters.multifeature_to_HUCs(feature=None, relationship='HAVE_THEIR_CENTER_IN')
base function that when provided with a feature class with multiple features
in it and a column containing the species designation will return a multipart feature for each species
usage: multifeature_to_HUCS(feature_class_location)
returns: an array of objects, sorted by species with a .zone attribute and attributes for every field in the data table.
PISCES.input_filters.pull_alt_codes()
PISCES.input_filters.pull_field_maps()
PISCES.input_filters.reproject_coords(rows, xaxis, yaxis, projection_file)
PISCES.input_filters.retrieve_if_methods()
PISCES.input_filters.table_to_array(dataset_path, transfer_object=False)

somewhere along the path of creating the set of functions that this function supports, we went the roundabout way. It is likely possible that simply creating a point at the x,y specified in the column and assigning the .shape attribute the the point would work. Even if that didn’t work, it’s still very possible that a more direct conversion of the table to a shape would work. For now, this code functions well and is pretty robust so I’m not willing to remove this code, but note that a potentially serious speedup may be possible through a less roundabout method (we arrived here as requirements changed while coding)

PISCES.input_filters.tablecoords_to_file(dataset_path, field_map, projection_index)

local_vars Module

exception PISCES.local_vars.DataProcessingError(value=None, set_index=None)

Bases: exceptions.Exception

exception PISCES.local_vars.DataStorageError(value=None)

Bases: exceptions.Exception

exception PISCES.local_vars.GenError(value)

Bases: exceptions.Exception

exception PISCES.local_vars.IFError(value)

Bases: exceptions.Exception

exception PISCES.local_vars.MappingError(value=None)

Bases: exceptions.Exception

PISCES.local_vars.data_setup()

Main function calling various others to set up internal variables that speed up PISCES

PISCES.local_vars.determine_delimiters()
class PISCES.local_vars.fish
PISCES.local_vars.get_version()
Just a compatibility function
Returns:
PISCES.local_vars.initialize()
Just an alias for data_setup
Returns:
class PISCES.local_vars.input_data(i1=None, i2=None, i3=None, i4=None, i5=None, i6=None, i7=None, i8=None, i9=None, i10=None, i11=None)
class PISCES.local_vars.observation
insert(db_cursor)
If we have data, then insert it into the database as a new record, returning the record id and setting it as an attribute. Use builtin funcs for this, passing a dictionary
Parameters:db_cursor
insert_collections(db_cursor)
Parameters:db_cursor
Returns:
load(record_id, from_table=None, db_cursor=None)

Given an observation id, loads the relevant observation from the database into this object use from_table to override the table to use (ie, to make an Invalid_Observations object. If db_cursor is provided, it will be used for the database. Otherwise, a new connection will be made.

class PISCES.local_vars.observation_set(l_dataset_path=None, l_dataset_name=None)
check_species()
cleanup_mark_input_row_imported()
insert_collections(db_cursor)
insert_data(db_cursor)
make_input_filter()
new_db_obs_set()
process_data()
record_data(db_cursor, db_conn)
save_data(db_cursor)
setup(dataset_key)
PISCES.local_vars.print_workspace_vars()
PISCES.local_vars.projection_code_to_file(projection_code)
class PISCES.local_vars.result_set(p_observations=[], p_if_method=None)
PISCES.local_vars.set_workspace_vars(l_workspace=None)
PISCES.local_vars.start(arc_script=False)

log Module

PISCES.log.bug(items)
PISCES.log.data_write(log_string)
PISCES.log.debug(log_string, screen=False)
PISCES.log.error(log_string)
PISCES.log.get_datestring()
PISCES.log.initialize(log_string=None, arc_script=0, log_file=None)
PISCES.log.warning(log_string)
PISCES.log.write(log_string, auto_print=0)

mapping Module

PISCES.mapping.begin(fish_subset, return_maps=False)
PISCES.mapping.cache_layer(arc_layer, bind_var, query_id, map_layer_object, db_cursor, force_search=False)
PISCES.mapping.check_common_layers(layer)
PISCES.mapping.clean_common_layers()
class PISCES.mapping.custom_query(query=None, query_rank=None, query_id=None, template_layer_file=None, callback=None, callback_arguments=None, layer_name=None, metadata_plugin=None, metadata_args=None, name_formula=None)
process_bind()
process_vars()
class PISCES.mapping.fish_map(map_title=None, short_name=None, query_set=None, query_set_name=None, base_mxd=None, base_ddp_mxd=None, callback=None, callback_args=None, name_formula=None)
choose_mxd(ddp)
clear_composed_layers()
decrement_ref_counts()
export_maps(mxd, file_only, ddp, extent_object, data_frame)
export_web_layers()
find_reference_layer(mxd)
generate(ddp=False)
get_custom_queries(db_conn)
get_data_sources_string()
If the map document has records (configured via a metadata plugin, return a string of the data sources
Returns:str: Data sources for the map
has_contents()
has_name_formulas()
We want to know if any of the layers have
Returns:
has_observations_records()
returns True if at least one of the layers has its observations records
Returns:
layers_visible(check_extent, data_frame)
make_layers(zone_layer, db_cursor, cache_layer_flag=True, unique_layer_name=False)
populate(db_cursor)
replace_variables(item, fill_spaces=False)
The goal of abstracting this out was that it was supposed to be used for query replacement too, but for now that would require too much additional re-engineering.
Parameters:item – string to run replacement operations on
Returns:string with replacement variables replaced
set_text_variables(mxd)
setup(queries, title=None, short_name=None, query_set=None, set_name=None, mxd=None, ddp_mxd=None, callback=None, callback_args=None, fish=None, name_formula=None)
Used to load the major class items later - __init__ contains some of the critical items, but most of it is set here. When provided custom_query object, sets up a map with the appropriate map_layer objects and custom queries
Parameters:
  • queries – a list (or other iterable) of custom_query objects
  • title – The title for the map. Accepts parameter placeholders.
  • short_name
  • query_set
  • set_name
  • mxd
  • ddp_mxd
  • callback
  • callback_args
  • fish
  • name_formula
Returns:

track_extent(extent_object, layer)
PISCES.mapping.generate_layer_name(bind_var, query_id, map_layer_object)
PISCES.mapping.get_bind_values(bind_column, db_cursor)
Parameters:
  • bind_column
  • db_cursor
Returns:

raise local_vars.MappingError:
 

PISCES.mapping.get_enabled_query_sets(db_cursor, set_ids)
PISCES.mapping.get_fish_to_map(fish_subset, db_cursor)
PISCES.mapping.get_mappable_fish(db_cursor, native_only=False, group=None)
Parameters:
  • db_cursor
  • native_only
  • group
Returns:

PISCES.mapping.get_multi_fish_query(query_set, db_conn, bind_column=None)
PISCES.mapping.initialize_maps(db_conn, set_ids=())
class PISCES.mapping.map_layer(query=None, bind_v=None, parent=None, text_query=None)
check_layer_cache(db_cursor, force_check=False)
Given a cursor, checks to see if there’s already a layer stored in the cache for this map_layer. If one is found, it sets self.cache_file to the value and returns True. Otherwise, returns False
Parameters:
  • db_cursor – a cursor to access the database
  • force_check – boolean: By default, this method only checks when the –continue flag is set on the command line, or when local_vars.continue_mapping is true. If force_check is True, this method checks for a cached layer regardless
Return boolean:

returns True when a layer is found and False when not

has_observations()
make(zone_layer=None, db_cursor=None)
Does the actual work of generating the spatial layer from the query. We should have already executed the query to retrieve the zones, but not retrieved the zones as a layer. This function batches a selection query to retrieve HUC12s from the zones layer, then export it out after optionally calling a callback. With the current query layer functionality in ArcMap, we could theoretically do a join (select from HUC12FullState, Observations where Observations.Zone_ID = HUC12FullState.HUC_12 and ...) instead of multiple selections, which would be much faster, but wouldn’t give us preexported layers. Some drawbacks and needs testing
Parameters:
  • zone_layer
  • db_cursor
Returns:

populate(db_cursor)
PISCES.mapping.mapping_process_config()
PISCES.mapping.refresh_zones()
Creates a feature layer from the HUC 12s in the database
Returns:str: Name of the feature layer created for mapping :raise local_vars.MappingError:
PISCES.mapping.replace_fid_with_common(replace_string)

‘a complicated way to make output occur with common name instead of FID - replaces the FID in the string rather than starting with common name earlier. Avoids problems with GDBs

PISCES.mapping.setup()
PISCES.mapping.structure_args(args)
PISCES.mapping.wrapup()
PISCES.mapping.write_csv_row(writer=None, fields=('fid', 'all_zip', 'kmz_1', 'kmz_2', 'kmz_16', 'kmz_17', 'kmz_18', 'kmz_48', 'lyr_1', 'lyr_2', 'lyr_16', 'lyr_17', 'lyr_18', 'lyr_48', 'shp_1', 'shp_2', 'shp_16', 'shp_17', 'shp_18', 'shp_48', 'png_map'), file=None, data=None)

observations Module

PISCES.observations.check_observations_db()
PISCES.observations.get_observation_certainties()

script_tool_funcs Module

This module contains a suite of functions relevant to the loading and running of the ArcGIS Toolbox tools for PISCES. While these tools may use functions in the broader set of PISCES code, this code tends to be specific to cross-tool needs within the PISCES toolbox. Examples would be code to set up and process the species and groups control, code that loads and sets up the observation types picker control, etc.

class PISCES.script_tool_funcs.Filter_Tool_Template

Bases: object

The Filter_Tool_Template class was meant to be a first crack at a common suite of code for a 10.1 and above compatible Python toolbox for PISCES. Due to time constraints, we haven’t migrated the PISCES code to a pure Python toolbox, but this code would be the initial suite of code that reduces redundancy and sets up its own tools when they have species pickers, etc. Still not fully functional, and needs work, but worth keeping for future development.

execute(parameters, messages)

The source code of the tool.

filter_params()

Define parameter definitions

isLicensed()

Set whether tool is licensed to execute.

updateMessages(parameters)

Modify the messages created by internal validation for each tool parameter. This method is called after internal validation.

updateParameters(parameters)

Modify the values and properties of parameters before internal validation is performed. This method is called whenever a parameter has been changed.

PISCES.script_tool_funcs.add_selection(val_class, selector_index, storage_index, objects_passed=False)
PISCES.script_tool_funcs.autocomplete_full_field(obj, field_num=1, objects_passed=False)
Parameters:
  • obj – the validation object from arcgis
  • field_num – the validator class params index of the field to autocomplete
  • objects_passed – If True, then you can pass the actual params object (instead of the index) in the field_num parameter. When False, the index must be passed
Returns:

boolean indicating whether a single, valid species is currently selected

PISCES.script_tool_funcs.deactivate_map(document='CURRENT')
A wrapper around turn_off_all_layers that dots our Is and crosses our Ts - makes sure that we check if we’re running in ArcMap or not, etc
Parameters:document – path of map document to operate on, or the keyword “CURRENT” as used in arcpy.mapping
Returns:2 items. dict index of visible layers, and a boolean value indicating whether the code is running within arcmap (ie, whether CURRENT is valid, if it was tried
PISCES.script_tool_funcs.determine_default_IF_for_addition(username)
Treating the username as an input filter code, it retrieves the best IF, raising ValueError if an Input Filter is not found. This should be caught and set the box to no value on the handling end.
Parameters:username – the PISCES username of the current install
Returns:
PISCES.script_tool_funcs.get_collections_filter()
PISCES.script_tool_funcs.get_fish_and_groups_filter(prepend_blank=True, sep='----', include_all=True)
Parameters:
  • prepend_blank – Boolean. Adds a ” ” at the beginning to be selectable
  • sep – a separator between the groups and the fish
Returns:

PISCES.script_tool_funcs.get_fish_filter(prepend=None)
In cases where we just want the fish filter (as opposed to fish and groups), this function provides just a listing of species codes and species
Parameters:prepend – list of items to appear at the beginning of the picker
Returns:
PISCES.script_tool_funcs.get_generic_filter_picker(table_column, validation_class, selector_index)
PISCES.script_tool_funcs.get_groups_filter()
PISCES.script_tool_funcs.get_input_filter_picker(validation_class, selector_index)
Parameters:
  • validation_class – the arcmap tool validation class object
  • selector_index – the index of the field we’re adding the filter list for in the validation class
Returns:

PISCES.script_tool_funcs.get_location()
This function is required primarily for script tools, but can be useful in other contexts too (it’s embedded in the code to set up workspaces as well.) For ArcGIS script tools, scripts run with no current working directory, so determining where the PISCES data is requires a global way to look it up. This function provides that by looking up the registry key SoftwareCWSPISCESlocation (note that on 64 bit computers running 32 bit Python, the key is at SoftwareWow6432NodeCWSPISCESlocation. This key contains the absolute path of the root PISCES folder. This function then checks to see if the current running copy of PISCES already knows where that base folder is, and if it doesn’t sets up the PISCES workspace (local_vars.set_workspace_vars()) with the retrieved value before returning the value to the caller.
Returns:
PISCES.script_tool_funcs.get_transactions_picker(validation_class, selector_index, minimum_transaction=2027)
Parameters:
  • validation_class – the arcmap tool validation class object
  • selector_index – the index of the field we’re adding the filter list for in the validation class
  • minimum_transaction – the lowest transaction ID to show in the picker. Transactions older than 2027 aren’t reversible since records weren’t associated with transactions at that point.
Returns:

None - sets the value on the validation class parameter

PISCES.script_tool_funcs.html_add_tablerow(items)
PROBABLY DEPRECATED along with the old Look Up Records tool. TODO: We should check for usages
Parameters:items
Returns:
PISCES.script_tool_funcs.is_in_arcgis(arcmap_only=False)

For purposes of testing, we need to sometimes check whether or not a tool is being run by ArcGIS or standalone. Allows script tools to return items to the table of contents unless we are running unit tests outside of ArcMap.

Parameters:arcmap_only – boolean. Default is False. When set to true, only detects whether the interpreter is running

in arcmap, otherwise, ArcCatalog is allowed as part of the test as well. Will likely need updating for ArcGIS Professional code.

PISCES.script_tool_funcs.load_map_sets(username='CWS')

Loads the currently available maps in the database - just the attributes Set_Name, Map_Title, and Set_Description. Returns a list of objects with those attributes.

Parameters:username – a string indicating the username to bring up maps for. Not a security issue, but a convenience issue. “CWS” shows all maps, and others may show a subset to not clutter the UI.
Returns:list of map objects, descriptive map string list
PISCES.script_tool_funcs.load_transaction(l_id='last', db_cursor=None)
PISCES.script_tool_funcs.make_collections_picker(validation_class, selector_index, storage_index)
Parameters:
  • validation_class – the arcmap tool validation class object
  • selector_index
  • storage_index
Returns:

PISCES.script_tool_funcs.make_config_backup()
Backups up the configuration file. Used when successfully loaded before editing it
Returns:
PISCES.script_tool_funcs.make_obs_type_picker(validation_class, selector_index)
PISCES.script_tool_funcs.make_species_group_picker(validation_class, selector_index, storage_index, objects_passed=False, include_all=True)
Parameters:
  • validation_class – the arcmap tool validation class object
  • selector_index
  • storage_index
  • objects_passed – indicates whether or not the index values passed in are actually the objects themselves (for new-style toolboxes)
Returns:

PISCES.script_tool_funcs.obs_type_selection_box_to_list(obs_type_as_string)
Given the output of the observation type picker box, returns the observation IDs
Parameters:obs_type_as_string
Returns:
PISCES.script_tool_funcs.parse_input_filter_picker(choice)
Corollary to get_input_filter_picker. Given a choice from it, it parses it and returns the input filter id, filter code, and name as a tuple
Parameters:choice – string choice from the picker created by get_input_filter_picker
Returns:tuple (id, filter code, name)
PISCES.script_tool_funcs.parse_input_species_from_list(l_search_species)

takes a species from an ArcGIS 10 Script tool dropdown in the form of “Code - Common Name” and parses out the code and returns it :param str l_search_species: species listing from script tool :return: str FID

PISCES.script_tool_funcs.parse_multi_species_group_list(l_species_and_groups)
Parameters:l_species_and_groups
Returns:
PISCES.script_tool_funcs.parse_obs_type_from_list(l_search_string)

takes an observation code from an ArcGIS 10 Script tool dropdown in the form of “Code - Description” and parses out the code and returns it

PISCES.script_tool_funcs.parse_transactions_picker(choice)
Corollary to get_transactions_picker Given a choice from it, it parses it and returns the transaction id and message as a tuple
Parameters:choice – string choice from the picker created by get_transactions_picker
Returns:tuple (id, message)
PISCES.script_tool_funcs.query_datasets_by_zone(zones, db_conn, species=None)
In process of deprecation: replaced by the Look Up Records tool, which no longer calls this function. Given a set of HUC_12s and, optionally, a species to limit them to, queries the observations table to determine what the data sources are.
Parameters:
  • zones
  • db_conn
  • species
Returns:

PISCES.script_tool_funcs.reactivate_map(document, activate_layers, running_in_arcmap)
A wrapper around turn_on_layers that checks the output of deactivate_map to see if we need to do anything
Parameters:
  • document
  • running_in_arcmap
  • activate_layers
Returns:

PISCES.script_tool_funcs.reverse_transaction(transaction_id, db_cursor)
PISCES.script_tool_funcs.set_default_IF_for_addition(username, validation_class, selector_index)
Sets the default input filter for the Add or Modify Data tool based upon the user’s username (looks up the input filter associated with it, and leaves it blank otherwise)
Parameters:
  • username – the PISCES username to use to find the input filter
  • validation_class – script tool validation class object
  • selector_index – the index order of the field with the input filter selector
Returns:

PISCES.script_tool_funcs.split_collection_names(picker_output)
Parameters:picker_output
Returns:
PISCES.script_tool_funcs.switch_iterator_field(new_table_field, old_table_field, validation_class, picker_index, storage_index)
When the mapping tool switches maps, the possible values to pass as arguments might change. This function handles the switching of those possible values and clears the existing selections when necessary and changes the selection list
Parameters:
  • new_table_field – the table and field (such as Species_Groups:FID) to pull values from
  • old_table_field – the current table and field in the same format (to see if it changed
  • validation_class – the mapping tool’s validation class
  • picker_index – the index of the picker field in the params list
  • storage_index – the index of the value storage field in the params list
Returns:

None - modifications are made in place by the tool

PISCES.script_tool_funcs.turn_off_all_layers(document='CURRENT')

A speedup function for map generation in ArcMap - turns off all layers so that it doesn’t try to rerender them while we’re using tools (since these tools need to run in the foreground and background processesing didn’t seem to speed it up).

Creates a dictionary keyed on the arcpy layer value longName which contains True or False values for whether or not the layers were enabled before running this. Allows us to then use turn_on_layers on the same document to reenable those layers

Parameters:document – a map document. defaults to “CURRENT”
Returns:dict: a dictionary keyed on layer longName values with True or False values for whether the layer was enabled.
PISCES.script_tool_funcs.turn_on_layers(document='CURRENT', storage_dict=None, only_change_visible=True)
PISCES.script_tool_funcs.validate_species(field)
PISCES.script_tool_funcs.write_table_from_observation_records(observations, return_gdb_table=True)
Given a set of PISCES orm_models.Observation instances, writes a CSV out for use in the Query HUC sources tool
Parameters:
  • observations – a PISCES orm set of observations
  • return_gdb_table – boolean: if True, returns a File Geodatabase Table. If False, returns the CSV name
Returns:

PISCES.script_tool_funcs.zones_feature_to_array(zones_layer)
WARNING: This function should be deprecated in favor of funcs.hucs_to_list
Parameters:zones_layer
Returns: