Insight network: Difference between revisions

From Opasnet
Jump to navigation Jump to search
(old code and table archived)
Line 352: Line 352:
Relation|referential link|edge.color|red|Referential link
Relation|referential link|edge.color|red|Referential link
Relation|referential link|edge.style|dashed|Referential link
Relation|referential link|edge.style|dashed|Referential link
</t2b>
==== Types of insight network tables ====
<t2b name="Table types" index="Type,Column names" obs="Dummy" unit="-">
oletus|type, Item, label, Relation, Object, Description, URL|1
sotearv|Ikaryhma, AHVK, Item, Teema, Ulottuvuus, Osiotyyppi, Tietolahde, KUVA id, Sotkanet id, JHS-luokka, Ryhman perustelut|1
hnh2035|Teema, Oldid, Item, Ohjelma, Vastuu, Aikajanne, Vaativuus, Kustannukset, Kust.kaupungille, Hyodyt.kaupungille, Kust.muille, Hyodyt.muille, Paastovahenema, Muut.vaikutukset, Seurantaindikaattori, Esimerkki, Description|1
kuvaind|Oldid, Item, Teema, Aihe2, Aihe3, Tietopaketti1, Tietopaketti2, Tietopaketti3, Mita.mittaa, Mitta-arvo ja muodostaminen, Tietolahde, Tietolahde ja tausta, Tuotannossa, Tietotarpeen taso, Tuottamistaso, Kansainvaliset tietotoimitukset, Muut kayttotarkoitukset, Description|1
sitra100|Item, Suuruus, Teema, URL|1
hvkertomus|Oldid, Item, Teema, Vaestoryhma, Lahde ja muodostaminen, Mita mittaa, KUVA-mittarissa, Hyte-kertoimessa, Arviointiraportissa, Hyvinvointikertomuksessa, Perustelut|1
hnhos|Oldid, type, Item, Description|1
keskustelu|Oldid, type, Item, label, Relation, Object, Description, URL|1
arviointi|Oldid, type, Item, Relation, Object, Description, URL, label|1
</t2b>
</t2b>


Line 669: Line 655:
</rcode>
</rcode>


<rcode name="chooseGr" label="Initiate ovariable chooseGr" embed=1>
# This is code Op_en3861/chooseGr on page [[Insight diagram]].
library(OpasnetUtils)
#' Function chooseGr takes a diagrammer graph and selects s subgraph based on topic, labels, steps from selected nodes etc.
#' @param gr diagrammer graph
#' @param input list of arguments to be used in selection
#' @seeds ovariable where @data has columns Topic to be chosen and Node for Oldid's to select.
#' @return diagrammer graph where node_selection contains the selected nodes
chooseGr <- function(gr, input, seeds=NULL, verbose=FALSE) {
  if(!is.null(seeds)) seeds <- match(seeds@data$Node[seeds@data$Topic==input$topic], gr$nodes_df$Oldid)
  nods <- union(c(
    seeds,
    match(input$addnodes, gr$nodes_df$label)),
    match(input$addnodesByid, gr$nodes_df$id)
  )
  nods <- nods[!is.na(nods)]
  gr <- deselect_nodes(gr, get_selection(gr))
  gr <- select_nodes_by_id(gr, nods)
  if(input$steps>0) {
    for(i in 1:input$steps) {
      gr <- deselect_nodes(gr,match(input$removenodes, gr$nodes_df$label))
      if(nrow(gr$node_selection)>0) {
        gr <- trav_both(gr,add_to_selection = TRUE)
      }
    }
    if("Remove branches only" %in% input$formatting) {
      gr <- select_nodes_by_id(gr,match(input$removenodes, gr$nodes_df$label))
    } else {
      gr <- deselect_nodes(gr,match(input$removenodes, gr$nodes_df$label))
    }
  }
  if("Show legend nodes" %in% input$formatting) {
    gr <- select_nodes_by_id(gr, match(seeds@data$Node[seeds@data$Topic=="Selitykset"], gr$nodes_df$Oldid))
  }
  gr <- deselect_nodes(gr, match(input$ignoreobj, gr$nodes_df$type))
  if(verbose) cat("Selected nodes: ", gr$nodes_df$label[gr$nodes_df$id %in% gr$node_selection[[1]]])
  return(gr)
}
objects.store(chooseGr)
cat("Ovariable chooseGr stored.\n")
</rcode>
==== Format tables ====


<rcode name="splizzeria" label="Initiate splizzeria (for developers only)" embed=1>
<rcode name="splizzeria" label="Initiate splizzeria (for developers only)" embed=1>
Line 730: Line 762:
objects.store(fillprev)
objects.store(fillprev)
cat("Function fillprev stored.\n")
cat("Function fillprev stored.\n")
</rcode>
<rcode name="grspec" label="Initiate grspec (for developers only)" embed=1>
# This is code Op_en3861/grspec on page [[Insight network]]
library(OpasnetUtils)
#' Updating graph properties
#'
#' grspec (graph specifier) replaces the default properties with the item-specific ones
#'
#' @param df data.frame whose properties are to be replaced
#' @param Parameter name of parameter as defined in prop_spec
#' @param prop_spec data.frame with specific properties
#' @return data.frame with the shape of df with an additional column whose name comes from Parameter
grspec <- function(df, Parameter, prop_spec) {
  out <- prop_spec[prop_spec$Parameter==Parameter,]
  coln <- ifelse("Object" %in% colnames(df), "Relation","Item")
  out <- out$Result[match(df[[coln]], out$Item)]
  return(as.character(out))
}
objects.store(grspec)
cat("Function grspec stored.\n")
</rcode>
<rcode name="makeInsightGraph" label="Initiate makeInsightGraph (for developers only)" embed=1>
# This is code Op_en3861/makeInsightGraph on page [[Insight network]]
library(OpasnetUtils)
#' Making insight network graph object
#'
#' makeInsightGraph is a function for fetching data for insight networks and making a graph object
#'
#' @param a is data.frame defining nodes and edges with at least columns: Oldid, type, Item, label, Relation, Object, Description. Other columns for nodes such as URL are allowed.
#' @return list of two data.frames: nodes_df and edges_df. These can be directly given as parameters for DiagrammeR::create_graph.
makeInsightGraph <- function(a) {
  require(OpasnetUtils)
  require(DiagrammeR)
 
  for(i in 1:ncol(a)) {
    a[[i]] <- gsub("[\"']", " ", a[[i]])
  }
 
  # First fetch the graphical styles of properties from [[Insight network]]
 
  prop_gen <- opbase.data(
    "Op_en3861", # [[Insight network]]
    subset="Graphical properties of objects and relations"
  )
  prop_gen <- splizzeria(prop_gen, c("Property","Value"))
 
  dbltst <- duplicated(prop_gen[c("Property","Value","Parameter")])
  if(any(dbltst)) {
    cat("There are double definitions in Insight networks/Graphical properties of objects and relations.\n")
    oprint(prop_gen[dbltst,])
  }
 
  ## Replace default setting with additional class info
 
  clas <- splizzeria(
    opbase.data("Op_en7783", subset="Hierarchies"), # [[Open policy ontology]]
    c("Class","Result")
  )[-1]
  # Clas for nodes (type)
  clas_n <- merge(a, clas, by.x="type", by.y="Result")
  clas_n <- unique(data.frame( # Currently looks at immediate, not recursive, hierarchies only.
    Item = rep(clas_n$Item,2),
    Class = c(clas_n$type,clas_n$Class),
    stringsAsFactors = FALSE
    #    clas_n[c("type","Item")],
    #    data.frame(type=clas_n$Class,Item=clas_n$Item)
  ))
  # Clas for edges (Relation)
  clas_e <- merge(a, clas, by.x="Relation", by.y="Result")
  clas_e <- unique( # Currently looks at immediate, not recursive, hierarchies only.
    data.frame(
      Relation=rep(clas_e$Relation,2),
      Class=c(clas_e$Relation,clas_e$Class),
      stringsAsFactors = FALSE
    )
  )
 
  # Create nodes 
 
  a$label <- ifelse(is.na(a$label), ifelse(is.na(a$Item), a$Oldid, substr(a$Item,1,30)), a$label)
 
  # Find Oldid that matches the Object based on Item, label, Oldid, or none (in this order)
  tmp  <- a$Oldid[match(a$Object, a$Item)]
  tmp2 <- a$Oldid[match(a$Object, a$label)]
  tmp  <- ifelse(is.na(tmp),tmp2,tmp)
  tmp2 <- a$Oldid[match(a$Object, a$Oldid)]
  tmp  <- ifelse(is.na(tmp), tmp2,tmp)
  tmp  <- ifelse(is.na(tmp), a$Object,tmp)
  a$Object <- tmp
 
  newbies <- setdiff(a$Object, a$Oldid)
  nodes <- orbind(
    a[!colnames(a) %in% c("Relation","Object")],
    data.frame(
      Oldid=newbies,
      Item=newbies,
      label=substr(newbies,1,30),
      stringsAsFactors = FALSE
    )
  )
 
  nodes <- nodes[!(duplicated(nodes$Oldid) | is.na(nodes$Item) | nodes$Item==""),]
#  nodes <- nodes[!(duplicated(nodes$Item) | is.na(nodes$Item) | nodes$Item==""),]
  nodes$id <- 1:nrow(nodes)
  nodes$tooltip <- paste0(
    nodes$label, ". ",
    ifelse(nodes$label == nodes$Item, "", paste0(nodes$Item, ". ")),
    ifelse(is.na(nodes$Description), "", paste0("\n", nodes$Description)),
    nodes$Description,
    " (", nodes$Oldid, "/", nodes$id,")"
  )
 
  # Create edges and flip unpreferred relations to their inverse relations
 
  inver <- opbase.data("Op_en7783", subset="Inverse relations")
 
  cols <- intersect(c("Oldid","Item","Relation","Object","Description","Edgedescription"),colnames(a))
  edges <- a[!(is.na(a$Object) | a$Object=="") , cols]
  flip <- edges$Relation %in% inver$Result
  tmp <- edges$Oldid
  edges$Oldid[flip] <- edges$Object[flip]
  edges$Object[flip] <- tmp[flip]
  edges$Relation[flip] <- as.character(inver$Preferred[match(edges$Relation, inver$Result)][flip])
  edges$id <- 1:nrow(edges)
  edges$from <- match(edges$Oldid, nodes$Oldid)
  edges$to <- match(edges$Object, nodes$Oldid)
  edges$rel<-edges$Relation # Not clear what rel is used for
  edges$label <- edges$Relation
  edges$labeltooltip <- paste0(edges$label, "(", edges$Oldid, "/", edges$id, ")")
 
  # Create a data.frame containing item-specific graph settings
 
  prop_spec <- merge(
    data.frame(Item = union(nodes$Item, edges$Relation)),
    prop_gen[prop_gen$Property=="default",c("Parameter","Result")]
  )
 
  # Replace default settings with property-specific settings
  for(i in 1:nrow(prop_gen)) {
    if(prop_gen$Property[i] != "default") {
      tst <- union( # Names of items that should have the property replaced
        nodes$Item[nodes[[prop_gen$Property[i]]]==prop_gen$Value[i]],
        c(edges$Relation[edges[[prop_gen$Property[i]]]==prop_gen$Value[i]],
          clas_n$Item[clas_n$Class == prop_gen$Value[i]],
          clas_e$Relation[clas_e$Class == prop_gen$Value[i]]
        )
      )
      prop_spec$Result[prop_spec$Item %in% tst & prop_spec$Parameter==prop_gen$Parameter[i]] <- prop_gen$Result[i]
    }
  }
 
  # Apply item-specific settings to nodes and edges
  nodes$fillcolor <- grspec(nodes,"node.fillcolor", prop_spec)
  nodes$color <- grspec(nodes,"node.color", prop_spec)
  nodes$fontcolor <- grspec(nodes,"node.fontcolor", prop_spec)
  nodes$fontsize <- as.numeric(grspec(nodes,"node.fontsize", prop_spec))
  nodes$shape <- grspec(nodes,"node.shape", prop_spec)
  nodes$sides <- as.numeric(grspec(nodes,"node.sides", prop_spec))
  nodes$width <- as.numeric(grspec(nodes, "node.width", prop_spec))
  nodes$height <- as.numeric(grspec(nodes,"node.height", prop_spec))
  nodes$penwidth <- as.numeric(grspec(nodes, "node.penwidth", prop_spec))
  nodes$orientation <- as.numeric(grspec(nodes,"node.orientation", prop_spec))
  nodes$skew <- as.numeric(grspec(nodes,"node.skew", prop_spec))
  nodes$distortion <- as.numeric(grspec(nodes, "node.distortion", prop_spec))
  edges$fontcolor <- grspec(edges,"edge.fontcolor", prop_spec)
  edges$penwidth <- grspec(edges,"edge.penwidth", prop_spec)
  edges$color <- grspec(edges,"edge.color", prop_spec)
  edges$style <- grspec(edges,"edge.style", prop_spec)
 
  return(
    create_graph(
      nodes_df=nodes,
      edges_df=edges
    )
  )
}
objects.store(makeInsightGraph)
cat("Function makeInsightGraph stored.\n")
</rcode>
<rcode name="chooseGr" label="Initiate ovariable chooseGr" embed=1>
# This is code Op_en3861/chooseGr on page [[Insight diagram]].
library(OpasnetUtils)
#' Function chooseGr takes a diagrammer graph and selects s subgraph based on topic, labels, steps from selected nodes etc.
#' @param gr diagrammer graph
#' @param input list of arguments to be used in selection
#' @seeds ovariable where @data has columns Topic to be chosen and Node for Oldid's to select.
#' @return diagrammer graph where node_selection contains the selected nodes
chooseGr <- function(gr, input, seeds=NULL, verbose=FALSE) {
  if(!is.null(seeds)) seeds <- match(seeds@data$Node[seeds@data$Topic==input$topic], gr$nodes_df$Oldid)
  nods <- union(c(
    seeds,
    match(input$addnodes, gr$nodes_df$label)),
    match(input$addnodesByid, gr$nodes_df$id)
  )
  nods <- nods[!is.na(nods)]
  gr <- deselect_nodes(gr, get_selection(gr))
  gr <- select_nodes_by_id(gr, nods)
  if(input$steps>0) {
    for(i in 1:input$steps) {
      gr <- deselect_nodes(gr,match(input$removenodes, gr$nodes_df$label))
      if(nrow(gr$node_selection)>0) {
        gr <- trav_both(gr,add_to_selection = TRUE)
      }
    }
    if("Remove branches only" %in% input$formatting) {
      gr <- select_nodes_by_id(gr,match(input$removenodes, gr$nodes_df$label))
    } else {
      gr <- deselect_nodes(gr,match(input$removenodes, gr$nodes_df$label))
    }
  }
  if("Show legend nodes" %in% input$formatting) {
    gr <- select_nodes_by_id(gr, match(seeds@data$Node[seeds@data$Topic=="Selitykset"], gr$nodes_df$Oldid))
  }
  gr <- deselect_nodes(gr, match(input$ignoreobj, gr$nodes_df$type))
  if(verbose) cat("Selected nodes: ", gr$nodes_df$label[gr$nodes_df$id %in% gr$node_selection[[1]]])
  return(gr)
}
objects.store(chooseGr)
cat("Ovariable chooseGr stored.\n")
</rcode>
==== Fetch data from the web: makeInsightTables function ====
<rcode name="makeInsightTables" label="Create function makeInsightTables (for developers only)" embed=1>
# This is code Op_en3861/makeInsightTables on page [[Insight network]]
library(OpasnetUtils)
#' @title makeInsightTables is a function for scraping web sources given in meta table and convert those to data.frames for further use.
#' @param meta table containing information about the tables, assessments, and discussions to scrape. Typically contains columns Name, Id, Type, URL, Table, Firstrow, Description.
#' @param types table with standardized column names that will be used. meta$Type matches with Type column. Other required columns: Column names.
#' @param savecsv TRUE if the created tables are to be saved as csv files. Default is FALSE.
#' @return a list of data.frames, including the original meta table as $meta.
makeInsightTables <- function(
  meta,
  types = opbase.data("Op_en3861", subset="Table types"),
  savecsv = FALSE
) {
  require(OpasnetUtils)
 
  # Call Op_fi5849/inaightNetwork first so you don't need these.
  #  objects.latest("Op_en3861",code_name="scrape.functions") # [[Insight network]] scrape.discussion .gssheet .webtable
  #  objects.latest("Op_en3861",code_name="scrape.assessment") # [[Insight network]] scrape.assessment
 
  if(!exists("insightTables")) insightTables <- list() # Existing insightTale can be updated partly or wholly
  if("ovariable" %in% class(meta)) meta <- meta@data # meta is originally an ovariable
  for(i in 1:ncol(meta)) meta[[i]] <- as.character(meta[[i]])
  for(i in c("Table","Firstrow")) meta[[i]] <- as.numeric(meta[[i]])
  meta$Name <- gsub("[Öö]","o",gsub("[ÄÅäå]","a",meta$Name))
 
  for(i in 1:nrow(meta)) {
    cat(i, meta$Name[i],"\n")
    if(grepl("google.com", meta$URL[i])) {
      out <- scrape.gssheet(meta$URL[i], meta$Firstrow[i])
    } else {
      if(meta$Type[i]=="keskustelu") {
        if(is.na(meta$Table[i])) j <- NULL else j <- as.numeric(meta$Table[i])
        out <- scrape.discussion(meta$URL[i], j)[[1]]
      } else {
        if(meta$Type[i] %in% c("arviointi","assessment")) {
          tmp <- strsplit(meta$URL[i], split="/")[[1]]
          objects.latest(tmp[1],tmp[2])
          dummy <- EvalOutput(get(tmp[3]))
          out <- scrape.assessment(get(tmp[3]))
          rm(dummy)
        } else {
          out <- scrape.webtable(meta$URL[i],meta$Table[i])
        }
      }
    }
    coln <- trimws(strsplit(as.character(types$`Column names`[types$Type==meta$Type[i]]),split=",")[[1]])
    if(colnames(out)[1]=="Obs") coln <- c("Oldid",coln)
    colnames(out) <- coln
    insightTables[[meta$Name[i]]] <- out
   
    #    if(savecsv) {
    #      wr#ite.csv(
    #        out,
    #        paste0(meta$Name[i],".csv"),
    #        quote=TRUE, row.names=FALSE, fileEncoding="UTF-8"
    #      )
    #    }
  }
 
  #  if(savecsv) {
  #    wr#ite.csv(meta, "meta.csv",fileEncoding = "UTF-8")
  #    # Zippaus ei jostain syystä toimi minun uudella koneella
  #    zip("~/Näkemysverkkojen tietotauluja.zip", paste0(c("meta",meta$Name),".csv"))
  #  }
  insightTables$meta <- meta
  return(insightTables)
}
objects.store(makeInsightTables)
cat("Function makeInsightTables stored.\n")
</rcode>
==== Insight network ====
This is an overall ovariable that automatically fetches all dependencies.
<rcode name="insightNetwork" label="Initiate insightNetwork ovariable (for developers only)" embed=1>
# This is code Op_en3861/insightNetwork on page [[Insight network]]
library(OpasnetUtils)
insightNetwork <- Ovariable(
  "insightNetwork",
  dependencies = data.frame(
    Name = c(
      "meta",
      "seeds",
      "objtypes",
      "makeInsightTables",
      "scrape.discussion","scrape.gssheet","scrape.webtable",
      "scrape.assessment",
      "makeGraphTable",
      "fillprev",
      "splizzeria",
      "grspec",
      "makeInsightGraph",
      "ui",
      "server"
    ),
    Ident = c(
      "Op_fi5849/meta",
      "Op_fi5849/seeds",
      "Op_fi5849/objtypes",
      "Op_en3861/makeInsightTables",
      rep("Op_en3861/scrape.functions",3), # These will be moved to OpasnetUtils package
      "Op_en3861/scrape.assessment",
      "Op_fi5849/makeGraphTable",
      "Op_en3861/fillprev",
      "Op_en3861/splizzeria",
      "Op_en3861/grspec",
      "Op_en3861/makeInsightGraph",
      "Op_en3861/ui",
      "Op_en3861/server"
    ),
    Description = c(
      "Contains metadata for the data tables and discussions to be fetched. Source: [[:op_fi:Näkemysverkko]]",
      "Seed nodes to be used as starting points for different graphs. Source: [[:op_fi:Näkemysverkko]]",
      "List of object types that you may deselect from the shiny graph. Source: [[:op_fi:Näkemysverkko]]",
      "Makes insight tables with minimal formatting. Source: [[Insight network]]",
      "Fetches data from a discussion", "Fetches data from a google sheet", "Fetches data from a regular webtable",
      "Fetches data from an assessment ovariable",
      "Converts insight tables to standard formatting. Source: [[:op_fi:Näkemysverkko]]",
      "Fills empty cells with content from a previous cell in that column. Source: [[Insight network]]",
      "Splits cell contents from a splitting character so that all other cells in that row multiply. Source: [[Insight network]]",
      "Updates generic graph formatting to object-specific formatting. Source: [[Insight network]]",
      "Creates DiagrammeR graph from data in standard structure. Source: [[Insight network]]",
      "Shiny user interface. Source: [[Insight network]]",
      "Shiny server code. Source: [[Insight network]]"
    )
  ),
  formula=function(...) {
    require(DiagrammeR)
    require(shiny)
    return(data.frame(Result=0))
  }
)
objects.store(insightNetwork)
cat("Ovariable insightNetwork stored.\n")
</rcode>
</rcode>


Line 1,192: Line 847:
library(OpasnetUtils)
library(OpasnetUtils)


#' Function chooseGr takes a diagrammer graph and selects s subgraph based on topic, labels, steps from selected nodes etc.
objects.latest("Op_en3861", code_name="chooseGr") # [[Insight network]] chooseGr
#' @param gr diagrammer graph
#' @param input list of arguments to be used in selection
#' @seeds ovariable where @data has columns Topic to be chosen and Node for Oldid's to select.
#' @return diagrammer graph where node_selection contains the selected nodes
 
chooseGr <- function(gr, input, seeds) {
  nods <- union(c(
    match(seeds@data$Node[seeds@data$Topic==input$topic], gr$nodes_df$Oldid),
    match(input$addnodes, gr$nodes_df$label)),
    match(input$addnodesByOldid, gr$nodes_df$Oldid)
  )
  nods <- nods[!is.na(nods)]
  gr <- deselect_nodes(gr, get_selection(gr))
  gr <- select_nodes_by_id(gr, nods)
  if(input$steps>0) {
    for(i in 1:input$steps) {
      gr <- deselect_nodes(gr,match(input$removenodes, gr$nodes_df$label))
      gr <- trav_both(gr,add_to_selection = TRUE)
    }
    if("Remove branches only" %in% input$formatting) {
      gr <- select_nodes_by_id(gr,match(input$removenodes, gr$nodes_df$label))
    } else {
      gr <- deselect_nodes(gr,match(input$removenodes, gr$nodes_df$label))
    }
  }
  if("Show legend nodes" %in% input$formatting) {
    gr <- select_nodes_by_id(gr, match(seeds@data$Node[seeds@data$Topic=="Selitykset"], gr$nodes_df$Oldid))
  }
  gr <- deselect_nodes(gr, match(input$ignoreobj, gr$nodes_df$type))
  return(gr)
}


#### Create shiny server at file server.R
#### Create shiny server at file server.R
Line 1,257: Line 881:


objects.store(chooseGr, server)
objects.store(chooseGr, server)
cat("Functions chooseGr, server stored. Usage: shinyApp(ui, server, enableBookmarking = 'url')\n")
cat("Functions chooseGr, server stored. Note! ChooseGr comes from its own code. Usage: shinyApp(ui, server, enableBookmarking = 'url')\n")
</rcode>
</rcode>


Line 1,825: Line 1,449:


== See also ==
== See also ==
* [http://en.opasnet.org/en-opwiki/index.php?title=Insight_network&oldid=42630 Arhived version] 15.1.2019 with several functionalities that are now depreciated and removed.
** T2b table [http://en.opasnet.org/w/Special:Opasnet_Base?id=op_en3861.table_types Table types] for different kinds of input tables.
** Code for function grspec. This is no longer needed as a generic formatted data.frame is used for formatting of all resources.
** Code for makeInsightGraph. This is replaced by makeGraph that has a better work flow.
** Code for makeInsightTables. Insighttables are no longer produced as they are replaced by context-specific ovariables that are on their respective knowledge crystal pages.
** Code for ovariable insightNetwork, which is an ovariable collecting all objects needed. Because of major updates, this is no longer useful.
** Code server: function chooseGr was updated and moved to an own code.


* [[Open policy practice]]
* [[Open policy practice]]

Revision as of 09:00, 15 January 2019


Insight networks are graphical representations of a particular situation, where the objects described are causally related to each other. In addition, the diagrams contain non-causal elements such as value judgements or inferences based on data. Insight networks utilise the ideas of directed acyclic graphs, but they have additional features.

Question

What notation is simple and flexible enough so that it can be used to represent all major issues related to a policy situation? It must be usable in both graphical and data formats.

Answer

+ Show code

Rationale

Process

The parameters of an argument and possible combinations.
Id Title Content Sign Target Type Paradigm Relation Result Comment
arg1234 Short title for display Actual argument Signature arg9876 relevance science attack 1 If paradigm changes (all else equal), relation may change, although typically only the result changes.
arg1234 Short title for display Actual argument Signature arg5555 relevance science comment 0
arg1234 Short title for display Actual argument Signature arg6666 truth science defense 1 Truth refers to the truth of the target
arg1234 Short title for display Actual argument Signature arg1234 selftruth science attack 0 Selftruth refers to the truth of the argument itself, unlike other types that refer to the target.
arg1234 Short title for display Actual argument Signature arg9876 relevance toldya comment 0
arg1234 Short title for display Actual argument Signature arg5555 relevance toldya defense 1
arg1234 Short title for display Actual argument Signature arg6666 truth toldya attack 0
arg1234 Short title for display Actual argument Signature arg1234 selftruth toldya comment 1 The relation in case of type=selftruth is irrelevant and is ignored.
These are unique to an argument
These are unique to an argument-target pair
These are unique to a triple of argument-target-paradigm
Insight networks have been described in a scientific article manuscript From open assessment to shared understanding: practical experiences#Insight networks. Objects and their relations used in open policy practice are described on page Open policy ontology.

There is a need for methods facilitating the flow of information and understanding between science and policy. The principle is to describe a risk situation in a formal manner. Insight networks contain items along a causal pathway (or network) from e.g. abatement strategies to emissions to dispersion to exposure to effects. They have been designed to describe also non-causal connections such as non-causal reasoning, values, preferences, and arguments.

These diagrams use graph theory with vertices (or nodes) and arcs (or arrows). They are used to describe and define all the pieces needed for a description of the situation under scrutiny. Diagrams may be produced with any graphics software, providing that calculation functions are not required. If calculations are needed, we recommend the use of R software and OpasnetUtils package.

Parameter properties
Parameter Css selector (Opasnet page scraping) Requirements
Id .argument attr=id Must start with a letter
Title .argument .title Short text. Is shown on insight graph as node label
Content .argument .content Text, may be long. Is shown with hover on graph
Sign .argument .sign a:first-of-type Must contain a link to participant's user page. Is shown with hover on graph
Target NA Previous argument one level up, or the statement for arguments on the first level
Type .argument i.type One of the three: relevance, truth, or selftruth (or "both", which is depreciated)
Paradigm .argument .paradigm Each paradigm should be described on a dedicated page. The rules implemented must be clear
Relation .argument .relation Is one of these: attack, defense, comment. "Branches" are typically uninteresting and ignored.
Result
  • relevance= .argument .relation attr=color. Gray= 0 (irrelevant), other=1 (relevant).
  • truth= .argument .relation attr=color. Gray=0 (untrue), other=1 (true)
  • selftruth= .argument .selftruth attr=color. Gray=0 (untrue), other=1 (true)
Truthlikeness of the relation. Either 1 or 0

This is the process how data flows into insight diagrams:

  • List of data tables of different insight diagrams is found from https://yhteistyotilat.fi/wiki08/x/1oGxAg. It has the following columns:
    • Ilmio: Name of the phenomenon. This will become the name of a csv data file.
    • Id: Identifier of the phenomenon. This will be used in Oldid of the items and relations.
    • Tyyppi: Type of the table. In practice, it defines the columns that the data table has. Different types are listed on #Types of insight network tables.
    • URL: Location of the data table. If the URL contains "google.com", it is assumed to be a google sheet. If the type (Tyyppi) is "keskustelu", it is assumed to be an Opasnet page with discussions. Otherwise, it is assumed to be a table on a web page that can be scraped with read_html() function.
    • Taulu: If the data is a table on a web page, it is the number of the table on that page. If the data is a discussion, it is the number of discussion; missing value means that all discussions on that page are read.
    • Alkurivi: In case of google sheets, it is the first row with actual data.
    • Kuvaus: Description of the table, with possible links to relevant description page.

All data tables and discussions are listed, formatted and saved as csv files in a zip file called op_fi:File:Näkemysverkkojen tietotauluja.zip. From there, the data can be accessed from within Opasnet Rtools. (The code scraping web pages does not work in Opasnet, although it is stored there.) Little formatting is done here, mainly the column titles are standardised. But the number and type of columns is not changed.

In the next phase, each csv file is opened, interpreted, and defined as items and relations. This is done in code Op_fi5810/graphs on page op_fi:Ympäristöterveysindikaattori. All these are saved as a DiagrammeR graph, and each topic may be separately selected as a subgraph.

Data

Graphical properties of objects and relations

Graphical properties of objects and relations(-)
ObsPropertyValueParameterResultDescription
1defaultdefaultnode.shapecircleDefault values unless something else is specified
2defaultdefaultnode.sides4
3defaultdefaultnode.skew0
4defaultdefaultnode.fillcolorwhite
5defaultdefaultnode.fontsize11
6defaultdefaultnode.height0.5
7defaultdefaultnode.width0.5
8defaultdefaultnode.colorbrown
9defaultdefaultnode.penwidth2
10defaultdefaultnode.fontcolorblack
11defaultdefaultnode.distortion0
12defaultdefaultedge.colorgrey
13defaultdefaultedge.fontsize10Not currently used
14defaultdefaultedge.fontcolorgrey
15defaultdefaultedge.styledotted
16defaultdefaultedge.penwidth2
17defaultdefaultedge.arrowsize1Not currently used
18typeunknownnode.fillcoloryellowThis formatting is used if there are undefined objects
19typeunknownnode.colorgreen
20typesubstancenode.shapecircleSubstantive type object
21typesubstancenode.fillcolorskyblue2Substantive type object
22typeknowledge crystalnode.colorgoldKnowledge crystal type object (including ovariables and key ovariables)
23typeoptionnode.colorpalevioletredOption for a decision
24typeoptionnode.fillcolorwhiteOption for a decision
25typeindexnode.shapepolygonIndex or other classifying determinant
26typeindexnode.sides4
27typeindexnode.skew0.5
28typeindexnode.fillcolorpurple1
29typeindexnode.height0.3
30typegraphnode.shapepolygonIndex or other classifying determinant
31typegraphnode.sides3
32typegraphnode.fillcolorpink
33typeassessmentnode.shapepolygonAssessment
34typeassessmentnode.sides8
35typeassessmentnode.fillcolorpurple1
36typestakeholdernode.shapehexagonStakeholder type object
37typestakeholdernode.fillcolorkhaki1Stakeholder type object
38typestakeholdernode.width0.8Stakeholder type object
39typemethodnode.shapepolygonMethod type object
40typemethodnode.sides6Method type object
41typemethodnode.fillcolorpurple1Method type object
42typeprocessnode.shapepentagonProcess type object
43typeprocessnode.fillcolorpurple1Process type object
44typeactionnode.fillcolorpinkProcess type object
45typeactionnode.shaperectangleDecision type object
46typetask 1node.colorbrownIllustration of the responsible organisation of the task
47typetask 2node.coloryellowIllustration of the responsible organisation of the task
48typetask 3node.colorblueIllustration of the responsible organisation of the task
49typetask 4node.colorgreenIllustration of the responsible organisation of the task
50typetask 5node.colorredIllustration of the responsible organisation of the task
51typedecisionnode.fillcolorredDecision type object
52typedatanode.shaperectangleData type object
53typedatanode.fillcolorgoldData type object
54typeobjectivenode.shapediamondObjective type object
55typeobjectivenode.fillcoloryellowObjective type object
56typeobjectivenode.width0.8Objective type object
57typepublicationnode.fillcolorgrayPublication type object
58typestatementnode.shapepolygonArgument type object
59typestatementnode.sides4Argument type object
60typestatementnode.width0.8Argument type object
61typestatementnode.distortion-0.5Argument type object
62typetrue statementnode.fillcolorgoldArgument type object
63typefalse statementnode.fillcolorgrayArgument type object
64typefact opening statementnode.fillcolorlightskyblue1Argument type object
65typevalue opening statementnode.fillcolorpalegreen1Argument type object
66typefact closing statementnode.fillcolorskyblueArgument type object
67typevalue closing statementnode.fillcolorspringgreenArgument type object
68typefact discussionnode.fillcolorskyblueArgument type object. Not neede?
69typevalue discussionnode.fillcolorspringgreenValue judgement type object. Not needed?
70typerisk factornode.colorpinkAdditional information about object class
71typeindicatornode.colorbrownAdditional information about object class
72typeindicatornode.fillcolorgoldAdditional information about object class
73typearviointikriteerinode.colororangeNot quite clear what criteria objects are: indicators or value statements, or something else
74typetasknode.colorgreenAdditional information about object class
75typedatanode.colororangeAdditional information about object class
76typehealth organisationnode.coloryellowAdditional information about object class
77Relationcausal linkedge.colorblackCausal link
78Relationcausal linkedge.stylesolidCausal link
79Relationpositive causal linkedge.fontcolorgreenCausal link
80Relationnegative causal linkedge.fontcolorredCausal link
81Relationparticipatory linkedge.colorpurpleParticipatory link
82Relationparticipatory linkedge.styledashedParticipatory link
83Relationoperational linkedge.colorblackOperational link
84Relationoperational linkedge.styledashedOperational link
85Relationevaluative linkedge.colorgreenEvaluative link
86Relationrelevant attackedge.colorredAttacking argument
87Relationrelevant defenseedge.colorgreenDefending argument
88Relationrelevant commentedge.colorblueCommenting argument
89Relationirrelevant argumentedge.colorgrayInvalid argument
90Relationargumentative linkedge.styledottedArgumentative link
91Relationargumentative linkedge.penwidth4Argumentative link
92Relationreferential linkedge.colorredReferential link
93Relationreferential linkedge.styledashedReferential link

Calculations

Insight network 2.0

An updated version should improve the

  • a) context sensitivity (referring to primarily to objects within own context but secondarily to those from another context),
  • b) making graphs by default from a single context rather than a full list of contexts from a meta table,
  • c) compatibility with cytoscape.js,
  • d) merging ready-made graphs meaningfully,
  • e) have a reasonable intermediate object format that contains all data needed, such as
    • tables for nodes and edges, compatible with Diagrammer, Cytoscape.js, AND Gephi.
    • metadata for display, such as seeds, steps, object types to ignore, whether to show labels etc. Or should these just be implemented on the graph?

What should be done?

  1. Fetch the data table by scrape or other function and with data about URL, table, and initial row.
  2. Use splizzeria and fillprev if needed.
  3. Interpret columns based on a vector of column numbers (with possibly 1+2 notation to paste columns) to create the standard columns. If this is done in an ovariable formula, there is no need for a specific function.
    • Context
    • Item
    • type
    • label
    • rel
    • Object
    • Description
    • Reldescription
    • URL
    • Result (dummy, always 0)
  4. Create missing node rows from objects. Do NOT assume context.
  5. Create URL from permanent resource location trunk and the identifier (where does the identifier come from?)
  6. Item ja label laitetaan pötköön ja haetaan mätsi. Tulos onrow-pötköstä.
  7. Create an ovariable from the table.
  8. Add meta to the ovariable with formatting data.
    • insightGraph:
      • seed
      • removenodes
      • formatting (character vector with possible entries: Hide node labels, Hide edge labels, Show legend nodes, Remove branches only)
      • ignoreobj
      • steps
  1. (NOT NEEDED? Create Oldid if does not exist from context and numbering)
  2. If a relation is presented as item, the formatting is applied to the ring.

Combine graph objects

  • Find items without context. Match them with items with the same Item (label) that do have a type.

Tuplarelaatiot, voidaanko kategorisesti poistaa?


Out <- rep(NA, length(find)) For(x in cond,) For(i in 1:length(find) Tmp<-id[context==contextfind(i))])[Match(find(i), df$cond(x)(df$context==contextfind(i))] pitää etsiä id alkuperäisestä taulukosta heti muuten ei toimi Out<- ifelse(isna(out). Tmp,out) )) Sitten sama ioman contekstirajoitusta.


Insight network 1.0

There are three different identifiers for a subject item.

  • Oldid: a technical identifier typically of format context.number, where number is a sequential number within a context.
  • Item: the actual name of the item, detailed enough to give a good understanding of its meaning.
  • label: a short name shown on insight networks. Does not exmplain everything, just enough to distinguish it from other items.

If Oldid is not given, it is created from the context and a number. If label is not given in data, it is truncated from Item.

Object item has one column Object that may contain any of these. The priority is Item > label > Oldid > Object. The last option means that it is assumed that Object refers to a new item that is not mentioned in the Item column.

An insight network is produced in this order (last object mentioned first).

  1. gr: a diagrammer graph with all data and formatting for an insight network. Produced by makeInsightGraph.
  2. makeInsightGraph


Making insight graphs

+ Show code

+ Show code

+ Show code

Format tables

+ Show code

+ Show code

Shiny server

+ Show code

+ Show code

Scrape functions

These functions will be placed in the OpasnetUtils package. For now, it must be manually copied to your code.

+ Show code

+ Show code

Copy descriptions to ovariables

The function assessmentDescriptions scans through an assessment ovarible that has all relevant assessment objects as parents. Dependencies slot may also have additional information, such as the following.

  • Name: name of parent (obligatory)
  • Ident: Opasnet page identifier and code name where the parent object can be loaded (e.g. Op_en7748/hia). Note: This is typically the code for the whole assessment, not the individual codes for the objects.
  • Token: Token for the model run where the parent object can be loaded (e.g. xxNsLw5hWdM6xyYp)
  • Description: A short description about what the object is. This is typically shown when cursor hovers over the object on an online insight diagram.
  • Page: Opasnet page identifier for the object's knowledge crystal page, which contains the research question, answer, and description of the object, together with discussion, if any. Typically this is empty for ovariables, because this information can be found from ovariable@meta slot and there is no need to duplicate it here.
  • Child: An object to which this object links. This is typically needed for objects such as graphs and data.frames that do not contain this information in their own structure, unlike ovariables. The direction of a relation is away from this object because then this object is the subject in triple sentences and can be given other parameters as well in other columns. A typical sentence is "graph describes ovariable", but for illustrative purposes this is inversed on insight networks so that the arrow points from an ovariable to a graph ("ovariable is described by graph").
  • Other columns are allowed.

+ Show code

Old notation

⇤--#: . Look at the table below together with Open policy ontology and merge. Decide which things should be on this page and which should be on the other. --Jouni (talk) 06:55, 24 April 2018 (UTC) (type: truth; paradigms: science: attack)

Node type Object Colour code in Analytica Comments
General variable 8R3B (automatic) This is a deterministic function of the quantities it depends on.
Chance variable 11L4B (autom) This is a variable which is uncertain and uncontrollable (in a direct sense).
Data-driven variable 3R1B A general variable where the result is mostly driven by data (observations or literature).
Author judgement variable 4R2B A general variable where the result is mainly driven by author judgement (estimates with poor or no data).
Decision variable 9L3B This is the variable that a decision-maker has the power to control. The decision variable should always be at the top of the chain of causality, even if this is a subchain i.e. it should not have any parent variables. Essentially the decision variable should be regarded as a decision that has to be made; since many factors affect all decisions it is not (in the case of INTARESE) an efficient use of resources to attempt to model what leads a decision-maker to make his decision.
Indicator 1R3B (autom) This is a variable of special interest. One of the indicators in an assessment may be the quantitative criterion that you are trying to optimize.

A particularly important variable in relation to the interests of the intended users of the assessment output (i.e. it must be a means of effective communication of assessment results).

  • It must be in causal connection to the endpoints of the assessment and thus address causality throughout the full chain.
  • It should reflect the use/purpose of the assessment.
  • It should address and be adapted according to the target audience.
  • It should be the ‘leading component’ in the assessment process.
Value judgement variable 8L4B A preference or value that a person or a group assigns to a particular condition or state of the world.
Index (or dimension) 5R2B (autom) This identifies the dimensions of the variable to which it is linked. Note that these dimensions do not have to be numeric, but can also be classes etc.
Risk assessment 8R3B (autom)
Scope 6R1B The scope of the object
Conclusion 6L3B A conclusion of the risk assessment (Result/Conclusion attribute).
Module 6R3B (autom) A group of variables that are put together for illustrative or other practical reasons.
Data 2L3B (autom) Contents of the Definition/Data attribute of a variable. If the Result attribute of a variable is used as Data for another variable, the first variable is called a proxy, and this node is used in the diagram. If an arrow or line is drawn between these objects, it must be noticed that this is NOT a causal link but an inference link. The direction of the arrow would be from the proxy to the variable.
Argument 8R2B A piece of argumentation related to an object (variable, risk assessment, or class)
Formula 9L3B Contents of the Definition/Formula attribute of a variable.
Class 1L2B A class object (a set of objects that share a particular property).
Function 4R2B (autom) A special kind of class. The particular property that is shared contains a full description of the Scope and the Definition attributes with given parameters.
Causal arrow This states a causal relationship (or influence) of one variable onto another. Note that causal arrows can only exist between two arrows; any arrows to or from non-causal objects are non-causal inference arrows.
Non-causal arrow This states an inference relationship between two objects. This means that the object where the arrow starts from is in the Data attribute of the other object. It is thus used to infer something about the value of the result of the latter object. Either object can be a variable or a non-variable. Note that Analytica is only able to show one kind of arrows, so in some cases the nature of the arrow (causal or inference) must be concluded from the context.

Previous notations

Previous notation for insight networks. This version was optimised for Analytica use.

Insight networks have previously been called pyrkilo diagrams, extended causal diagrams, and factor-effect-value networks. These names are no longer in active use. An archived version of the notation can be found from an earlier version of this page.

See also

  • Arhived version 15.1.2019 with several functionalities that are now depreciated and removed.
    • T2b table Table types for different kinds of input tables.
    • Code for function grspec. This is no longer needed as a generic formatted data.frame is used for formatting of all resources.
    • Code for makeInsightGraph. This is replaced by makeGraph that has a better work flow.
    • Code for makeInsightTables. Insighttables are no longer produced as they are replaced by context-specific ovariables that are on their respective knowledge crystal pages.
    • Code for ovariable insightNetwork, which is an ovariable collecting all objects needed. Because of major updates, this is no longer useful.
    • Code server: function chooseGr was updated and moved to an own code.

References